Kubernetes 장애 점검 리포트

Pod: cert-manager-7ddd8cdb9f-z2xzm (Namespace: cert-manager)

Logs (filtered)

I0428 06:44:46.857190       1 controller.go:162] "re-queuing item due to optimistic locking on resource" logger="cert-manager.certificates-readiness" key="kubeflow/kfp-cache-cert" error="Operation cannot be fulfilled on certificates.cert-manager.io \"kfp-cache-cert\": the object has been modified; please apply your changes to the latest version and try again"
I0428 06:44:46.976572       1 controller.go:162] "re-queuing item due to optimistic locking on resource" logger="cert-manager.certificates-key-manager" key="kubeflow/kfp-cache-cert" error="Operation cannot be fulfilled on certificates.cert-manager.io \"kfp-cache-cert\": the object has been modified; please apply your changes to the latest version and try again"
I0428 06:44:47.092542       1 controller.go:162] "re-queuing item due to optimistic locking on resource" logger="cert-manager.certificates-readiness" key="kubeflow/kfp-cache-cert" error="Operation cannot be fulfilled on certificates.cert-manager.io \"kfp-cache-cert\": the object has been modified; please apply your changes to the latest version and try again"
I0428 06:44:47.107680       1 controller.go:162] "re-queuing item due to optimistic locking on resource" logger="cert-manager.certificates-key-manager" key="kubeflow/kfp-cache-cert" error="Operation cannot be fulfilled on certificates.cert-manager.io \"kfp-cache-cert\": the object has been modified; please apply your changes to the latest version and try again"
I0428 06:45:40.938508       1 controller.go:162] "re-queuing item due to optimistic locking on resource" logger="cert-manager.certificates-trigger" key="kubeflow/serving-cert" error="Operation cannot be fulfilled on certificates.cert-manager.io \"serving-cert\": the object has been modified; please apply your changes to the latest version and try again"
I0428 06:45:41.309535       1 controller.go:162] "re-queuing item due to optimistic locking on resource" logger="cert-manager.certificates-readiness" key="kubeflow/serving-cert" error="Operation cannot be fulfilled on certificates.cert-manager.io \"serving-cert\": the object has been modified; please apply your changes to the latest version and try again"
I0428 06:45:41.325735       1 controller.go:162] "re-queuing item due to optimistic locking on resource" logger="cert-manager.certificates-readiness" key="kubeflow/serving-cert" error="Operation cannot be fulfilled on certificates.cert-manager.io \"serving-cert\": the object has been modified; please apply your changes to the latest version and try again"
I0428 06:49:08.263224       1 controller.go:162] "re-queuing item due to optimistic locking on resource" logger="cert-manager.certificates-trigger" key="kubeflow/katib-webhook-cert" error="Operation cannot be fulfilled on certificates.cert-manager.io \"katib-webhook-cert\": the object has been modified; please apply your changes to the latest version and try again"
I0428 06:49:08.596783       1 controller.go:162] "re-queuing item due to optimistic locking on resource" logger="cert-manager.certificates-issuing" key="kubeflow/katib-webhook-cert" error="Operation cannot be fulfilled on certificates.cert-manager.io \"katib-webhook-cert\": the object has been modified; please apply your changes to the latest version and try again"
I0428 06:49:08.641065       1 controller.go:162] "re-queuing item due to optimistic locking on resource" logger="cert-manager.certificates-key-manager" key="kubeflow/katib-webhook-cert" error="Operation cannot be fulfilled on certificates.cert-manager.io \"katib-webhook-cert\": the object has been modified; please apply your changes to the latest version and try again"
I0428 06:49:16.700675       1 controller.go:162] "re-queuing item due to optimistic locking on resource" logger="cert-manager.certificates-readiness" key="kubeflow/admission-webhook-cert" error="Operation cannot be fulfilled on certificates.cert-manager.io \"admission-webhook-cert\": the object has been modified; please apply your changes to the latest version and try again"
I0428 06:49:16.928182       1 controller.go:162] "re-queuing item due to optimistic locking on resource" logger="cert-manager.certificates-key-manager" key="kubeflow/admission-webhook-cert" error="Operation cannot be fulfilled on certificates.cert-manager.io \"admission-webhook-cert\": the object has been modified; please apply your changes to the latest version and try again"
I0428 06:49:16.998387       1 controller.go:162] "re-queuing item due to optimistic locking on resource" logger="cert-manager.certificates-readiness" key="kubeflow/admission-webhook-cert" error="Operation cannot be fulfilled on certificates.cert-manager.io \"admission-webhook-cert\": the object has been modified; please apply your changes to the latest version and try again"
I0428 06:49:17.004264       1 controller.go:162] "re-queuing item due to optimistic locking on resource" logger="cert-manager.certificates-issuing" key="kubeflow/admission-webhook-cert" error="Operation cannot be fulfilled on certificates.cert-manager.io \"admission-webhook-cert\": the object has been modified; please apply your changes to the latest version and try again"
I0428 06:49:17.009986       1 controller.go:162] "re-queuing item due to optimistic locking on resource" logger="cert-manager.certificates-key-manager" key="kubeflow/admission-webhook-cert" error="Operation cannot be fulfilled on certificates.cert-manager.io \"admission-webhook-cert\": the object has been modified; please apply your changes to the latest version and try again"
I0428 06:49:32.870986       1 controller.go:162] "re-queuing item due to optimistic locking on resource" logger="cert-manager.certificates-trigger" key="kubeflow/pvcviewer-serving-cert" error="Operation cannot be fulfilled on certificates.cert-manager.io \"pvcviewer-serving-cert\": the object has been modified; please apply your changes to the latest version and try again"
I0428 06:49:33.392302       1 controller.go:162] "re-queuing item due to optimistic locking on resource" logger="cert-manager.certificates-key-manager" key="kubeflow/pvcviewer-serving-cert" error="Operation cannot be fulfilled on certificates.cert-manager.io \"pvcviewer-serving-cert\": the object has been modified; please apply your changes to the latest version and try again"
I0428 06:49:33.472544       1 controller.go:162] "re-queuing item due to optimistic locking on resource" logger="cert-manager.certificates-readiness" key="kubeflow/pvcviewer-serving-cert" error="Operation cannot be fulfilled on certificates.cert-manager.io \"pvcviewer-serving-cert\": the object has been modified; please apply your changes to the latest version and try again"
I0428 06:49:33.491700       1 controller.go:162] "re-queuing item due to optimistic locking on resource" logger="cert-manager.certificates-key-manager" key="kubeflow/pvcviewer-serving-cert" error="Operation cannot be fulfilled on certificates.cert-manager.io \"pvcviewer-serving-cert\": the object has been modified; please apply your changes to the latest version and try again"

Pod: cert-manager-cainjector-57cd76c845-smrfg (Namespace: cert-manager)

Logs (filtered)

E0428 06:49:33.664559       1 controller.go:329] "Reconciler error" err="Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io \"pvcviewers.kubeflow.org\": the object has been modified; please apply your changes to the latest version and try again" logger="cert-manager" controller="customresourcedefinition" controllerGroup="apiextensions.k8s.io" controllerKind="CustomResourceDefinition" CustomResourceDefinition="pvcviewers.kubeflow.org" namespace="" name="pvcviewers.kubeflow.org" reconcileID="b678024f-63fc-4c38-91fe-27b2611ec1e0"

Pod: cluster-local-gateway-595b55bdb4-kgj8w (Namespace: istio-system)

Logs (filtered)

2025-04-28T06:43:50.319069Z	info	FLAG: --proxyComponentLogLevel="misc:error"
2025-04-28T06:43:50.560230Z	info	Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --concurrency 2]

Pod: istio-ingressgateway-66c5cb4897-l2j4l (Namespace: istio-system)

Logs (filtered)

2025-04-28T05:53:01.512093Z	info	FLAG: --proxyComponentLogLevel="misc:error"
2025-04-28T05:53:03.549976Z	info	Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --concurrency 2]
2025-04-28T05:53:23.523589Z	warn	ca	ca request failed, starting attempt 1 in 97.412594ms
2025-04-28T05:53:23.622193Z	warn	ca	ca request failed, starting attempt 2 in 186.606322ms
2025-04-28T05:53:23.809744Z	warn	ca	ca request failed, starting attempt 3 in 432.04241ms
2025-04-28T05:53:24.242543Z	warn	ca	ca request failed, starting attempt 4 in 864.987303ms
2025-04-28T05:53:25.108554Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T05:53:25.109039Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T05:53:25.109047Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T05:53:43.539275Z	warn	ca	ca request failed, starting attempt 1 in 93.149173ms
2025-04-28T05:53:43.633195Z	warn	ca	ca request failed, starting attempt 2 in 201.662578ms
2025-04-28T05:53:43.835670Z	warn	ca	ca request failed, starting attempt 3 in 377.217092ms
2025-04-28T05:53:44.213393Z	warn	ca	ca request failed, starting attempt 4 in 834.552711ms
2025-04-28T05:53:45.048505Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:59603->10.96.0.10:53: i/o timeout"
2025-04-28T05:53:45.048989Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:59603->10.96.0.10:53: i/o timeout"
2025-04-28T05:53:45.048998Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:59603->10.96.0.10:53: i/o timeout"
2025-04-28T05:54:04.137533Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 40s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T05:54:05.049308Z	warn	ca	ca request failed, starting attempt 1 in 100.409743ms
2025-04-28T05:54:05.150678Z	warn	ca	ca request failed, starting attempt 2 in 199.560789ms
2025-04-28T05:54:05.351121Z	warn	ca	ca request failed, starting attempt 3 in 360.195264ms
2025-04-28T05:54:05.711594Z	warn	ca	ca request failed, starting attempt 4 in 740.748262ms
2025-04-28T05:54:06.453544Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T05:54:06.454035Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T05:54:06.454044Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T05:54:24.139938Z	warn	ca	ca request failed, starting attempt 1 in 92.87879ms
2025-04-28T05:54:24.140356Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 60s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:48614->10.96.0.10:53: i/o timeout"	thread=15
2025-04-28T05:54:24.233684Z	warn	ca	ca request failed, starting attempt 2 in 218.897009ms
2025-04-28T05:54:24.453280Z	warn	ca	ca request failed, starting attempt 3 in 400.339083ms
2025-04-28T05:54:24.853920Z	warn	ca	ca request failed, starting attempt 4 in 844.769584ms
2025-04-28T05:54:25.699064Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:48614->10.96.0.10:53: i/o timeout"
2025-04-28T05:54:25.699968Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:48614->10.96.0.10:53: i/o timeout"
2025-04-28T05:54:25.699988Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:48614->10.96.0.10:53: i/o timeout"
2025-04-28T05:54:45.702174Z	warn	ca	ca request failed, starting attempt 1 in 106.643513ms
2025-04-28T05:54:45.809620Z	warn	ca	ca request failed, starting attempt 2 in 185.982224ms
2025-04-28T05:54:45.996126Z	warn	ca	ca request failed, starting attempt 3 in 414.138797ms
2025-04-28T05:54:46.410772Z	warn	ca	ca request failed, starting attempt 4 in 795.524384ms
2025-04-28T05:54:47.124513Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 83s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T05:54:47.207074Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T05:54:47.207723Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T05:54:47.207737Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T05:55:05.707538Z	warn	ca	ca request failed, starting attempt 1 in 109.561496ms
2025-04-28T05:55:05.707901Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 102s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:54228->10.96.0.10:53: i/o timeout"	thread=15
2025-04-28T05:55:05.818245Z	warn	ca	ca request failed, starting attempt 2 in 191.517891ms
2025-04-28T05:55:06.010741Z	warn	ca	ca request failed, starting attempt 3 in 370.313842ms
2025-04-28T05:55:06.381324Z	warn	ca	ca request failed, starting attempt 4 in 832.262359ms
2025-04-28T05:55:07.213918Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:54228->10.96.0.10:53: i/o timeout"
2025-04-28T05:55:07.214513Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:54228->10.96.0.10:53: i/o timeout"
2025-04-28T05:55:07.214522Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:54228->10.96.0.10:53: i/o timeout"
2025-04-28T05:55:27.217351Z	warn	ca	ca request failed, starting attempt 1 in 104.884842ms
2025-04-28T05:55:27.322968Z	warn	ca	ca request failed, starting attempt 2 in 219.822839ms
2025-04-28T05:55:27.543472Z	warn	ca	ca request failed, starting attempt 3 in 387.383384ms
2025-04-28T05:55:27.931078Z	warn	ca	ca request failed, starting attempt 4 in 809.589278ms
2025-04-28T05:55:28.741348Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T05:55:28.741834Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T05:55:28.741843Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T05:55:32.506748Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 128s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T05:55:47.224564Z	warn	ca	ca request failed, starting attempt 1 in 100.594583ms
2025-04-28T05:55:47.325993Z	warn	ca	ca request failed, starting attempt 2 in 213.17758ms
2025-04-28T05:55:47.539435Z	warn	ca	ca request failed, starting attempt 3 in 435.491629ms
2025-04-28T05:55:47.975101Z	warn	ca	ca request failed, starting attempt 4 in 869.833341ms
2025-04-28T05:55:48.845848Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:46838->10.96.0.10:53: i/o timeout"
2025-04-28T05:55:48.846445Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:46838->10.96.0.10:53: i/o timeout"
2025-04-28T05:55:48.846459Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:46838->10.96.0.10:53: i/o timeout"
2025-04-28T05:56:08.847542Z	warn	ca	ca request failed, starting attempt 1 in 90.816899ms
2025-04-28T05:56:08.939202Z	warn	ca	ca request failed, starting attempt 2 in 205.956889ms
2025-04-28T05:56:09.145603Z	warn	ca	ca request failed, starting attempt 3 in 402.792224ms
2025-04-28T05:56:09.549344Z	warn	ca	ca request failed, starting attempt 4 in 775.596623ms
2025-04-28T05:56:10.325301Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T05:56:10.325783Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T05:56:10.325792Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T05:56:18.414164Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 174s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T05:56:28.851669Z	warn	ca	ca request failed, starting attempt 1 in 93.248088ms
2025-04-28T05:56:28.946049Z	warn	ca	ca request failed, starting attempt 2 in 218.303352ms
2025-04-28T05:56:29.165474Z	warn	ca	ca request failed, starting attempt 3 in 423.577623ms
2025-04-28T05:56:29.590210Z	warn	ca	ca request failed, starting attempt 4 in 832.858725ms
2025-04-28T05:56:30.423468Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:40362->10.96.0.10:53: i/o timeout"
2025-04-28T05:56:30.424008Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:40362->10.96.0.10:53: i/o timeout"
2025-04-28T05:56:30.424018Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:40362->10.96.0.10:53: i/o timeout"
2025-04-28T05:56:59.321755Z	warn	ca	ca request failed, starting attempt 1 in 102.373505ms
2025-04-28T05:56:59.425090Z	warn	ca	ca request failed, starting attempt 2 in 202.544433ms
2025-04-28T05:56:59.628580Z	warn	ca	ca request failed, starting attempt 3 in 374.148132ms
2025-04-28T05:57:00.003140Z	warn	ca	ca request failed, starting attempt 4 in 814.578417ms
2025-04-28T05:57:00.818646Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T05:57:00.819214Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T05:57:00.819223Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T05:57:07.191138Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 223s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T05:57:31.177677Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 247s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T05:57:36.233795Z	warn	ca	ca request failed, starting attempt 1 in 108.869365ms
2025-04-28T05:57:36.343333Z	warn	ca	ca request failed, starting attempt 2 in 208.941979ms
2025-04-28T05:57:36.553085Z	warn	ca	ca request failed, starting attempt 3 in 403.834477ms
2025-04-28T05:57:36.957897Z	warn	ca	ca request failed, starting attempt 4 in 743.270652ms
2025-04-28T05:57:37.702138Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T05:57:37.702782Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T05:57:37.702798Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T05:57:51.189527Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 267s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:58739->10.96.0.10:53: i/o timeout"	thread=15
2025-04-28T05:58:23.959369Z	warn	ca	ca request failed, starting attempt 1 in 95.926306ms
2025-04-28T05:58:24.055795Z	warn	ca	ca request failed, starting attempt 2 in 187.730521ms
2025-04-28T05:58:24.244324Z	warn	ca	ca request failed, starting attempt 3 in 433.550601ms
2025-04-28T05:58:24.679116Z	warn	ca	ca request failed, starting attempt 4 in 722.364494ms
2025-04-28T05:58:25.402120Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T05:58:25.402703Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T05:58:25.402713Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T05:58:36.067281Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 312s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T05:58:53.547737Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 329s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:48179->10.96.0.10:53: i/o timeout"	thread=15
2025-04-28T05:59:16.000940Z	warn	ca	ca request failed, starting attempt 1 in 101.135445ms
2025-04-28T05:59:16.102195Z	warn	ca	ca request failed, starting attempt 2 in 196.569873ms
2025-04-28T05:59:16.299678Z	warn	ca	ca request failed, starting attempt 3 in 373.129627ms
2025-04-28T05:59:16.673324Z	warn	ca	ca request failed, starting attempt 4 in 753.42613ms
2025-04-28T05:59:17.427431Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T05:59:17.428095Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T05:59:17.428105Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T05:59:33.059155Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 369s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:00:00.857566Z	warn	ca	ca request failed, starting attempt 1 in 106.238295ms
2025-04-28T06:00:00.965029Z	warn	ca	ca request failed, starting attempt 2 in 200.261447ms
2025-04-28T06:00:01.165686Z	warn	ca	ca request failed, starting attempt 3 in 380.051156ms
2025-04-28T06:00:01.546379Z	warn	ca	ca request failed, starting attempt 4 in 832.040735ms
2025-04-28T06:00:02.379487Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:00:02.380005Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:00:02.380014Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:00:08.890053Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 405s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:00:47.062617Z	warn	ca	ca request failed, starting attempt 1 in 102.903297ms
2025-04-28T06:00:47.165974Z	warn	ca	ca request failed, starting attempt 2 in 207.190979ms
2025-04-28T06:00:47.373327Z	warn	ca	ca request failed, starting attempt 3 in 435.483482ms
2025-04-28T06:00:47.808910Z	warn	ca	ca request failed, starting attempt 4 in 775.300317ms
2025-04-28T06:00:48.585141Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:00:48.585618Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:00:48.585628Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:00:58.871131Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 455s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:01:18.578190Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 474s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:33552->10.96.0.10:53: i/o timeout"	thread=15
2025-04-28T06:01:45.286088Z	warn	ca	ca request failed, starting attempt 1 in 90.552172ms
2025-04-28T06:01:45.377434Z	warn	ca	ca request failed, starting attempt 2 in 184.696019ms
2025-04-28T06:01:45.562840Z	warn	ca	ca request failed, starting attempt 3 in 425.818698ms
2025-04-28T06:01:45.989509Z	warn	ca	ca request failed, starting attempt 4 in 734.388313ms
2025-04-28T06:01:46.724496Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:01:46.725045Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:01:46.725057Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:01:51.728789Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 508s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:02:25.418110Z	warn	ca	ca request failed, starting attempt 1 in 107.820267ms
2025-04-28T06:02:25.526493Z	warn	ca	ca request failed, starting attempt 2 in 196.260403ms
2025-04-28T06:02:25.722914Z	warn	ca	ca request failed, starting attempt 3 in 428.528133ms
2025-04-28T06:02:26.151742Z	warn	ca	ca request failed, starting attempt 4 in 832.018606ms
2025-04-28T06:02:26.984907Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:02:26.985439Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:02:26.985452Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:02:28.158663Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 544s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:03:17.589874Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 593s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:03:28.095385Z	warn	ca	ca request failed, starting attempt 1 in 107.973537ms
2025-04-28T06:03:28.203815Z	warn	ca	ca request failed, starting attempt 2 in 212.007087ms
2025-04-28T06:03:28.416358Z	warn	ca	ca request failed, starting attempt 3 in 403.301882ms
2025-04-28T06:03:28.820624Z	warn	ca	ca request failed, starting attempt 4 in 747.971068ms
2025-04-28T06:03:29.569662Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:51956->10.96.0.10:53: i/o timeout"
2025-04-28T06:03:29.570098Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:51956->10.96.0.10:53: i/o timeout"
2025-04-28T06:03:29.570108Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:51956->10.96.0.10:53: i/o timeout"
2025-04-28T06:03:48.717678Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 625s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:04:28.113137Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 664s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:04:40.571542Z	warn	ca	ca request failed, starting attempt 1 in 92.466955ms
2025-04-28T06:04:40.664891Z	warn	ca	ca request failed, starting attempt 2 in 182.009986ms
2025-04-28T06:04:40.847460Z	warn	ca	ca request failed, starting attempt 3 in 421.585933ms
2025-04-28T06:04:41.270201Z	warn	ca	ca request failed, starting attempt 4 in 847.737463ms
2025-04-28T06:04:42.118871Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:04:42.119343Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:04:42.119354Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:04:46.405428Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 682s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:60550->10.96.0.10:53: i/o timeout"	thread=15
2025-04-28T06:05:07.708963Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 704s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:05:27.414500Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 723s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:41051->10.96.0.10:53: i/o timeout"	thread=15
2025-04-28T06:05:51.251635Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 747s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:06:04.009502Z	warn	ca	ca request failed, starting attempt 1 in 98.959944ms
2025-04-28T06:06:04.108881Z	warn	ca	ca request failed, starting attempt 2 in 210.320753ms
2025-04-28T06:06:04.320371Z	warn	ca	ca request failed, starting attempt 3 in 414.739414ms
2025-04-28T06:06:04.736088Z	warn	ca	ca request failed, starting attempt 4 in 839.994294ms
2025-04-28T06:06:05.576346Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:06:05.576881Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:06:05.576891Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:06:10.121033Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 766s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:50586->10.96.0.10:53: i/o timeout"	thread=15
2025-04-28T06:06:58.381335Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 814s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:07:12.607525Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 828s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:53092->10.96.0.10:53: i/o timeout"	thread=15
2025-04-28T06:07:20.145722Z	warn	ca	ca request failed, starting attempt 1 in 109.267452ms
2025-04-28T06:07:20.256120Z	warn	ca	ca request failed, starting attempt 2 in 185.642646ms
2025-04-28T06:07:20.442548Z	warn	ca	ca request failed, starting attempt 3 in 411.900448ms
2025-04-28T06:07:20.855216Z	warn	ca	ca request failed, starting attempt 4 in 799.122695ms
2025-04-28T06:07:21.654589Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:53092->10.96.0.10:53: i/o timeout"
2025-04-28T06:07:21.655073Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:53092->10.96.0.10:53: i/o timeout"
2025-04-28T06:07:21.655081Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:53092->10.96.0.10:53: i/o timeout"
2025-04-28T06:07:37.399538Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 853s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:07:56.300097Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 872s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:35732->10.96.0.10:53: i/o timeout"	thread=15
2025-04-28T06:08:17.168885Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 893s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:08:21.911684Z	warn	ca	ca request failed, starting attempt 1 in 103.856025ms
2025-04-28T06:08:22.016108Z	warn	ca	ca request failed, starting attempt 2 in 209.64821ms
2025-04-28T06:08:22.227007Z	warn	ca	ca request failed, starting attempt 3 in 426.342023ms
2025-04-28T06:08:22.653666Z	warn	ca	ca request failed, starting attempt 4 in 729.404616ms
2025-04-28T06:08:23.383652Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:08:23.384233Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:08:23.384242Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:08:58.698454Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 935s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:09:18.195105Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 954s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:57677->10.96.0.10:53: i/o timeout"	thread=15
2025-04-28T06:09:23.709508Z	warn	ca	ca request failed, starting attempt 1 in 93.090741ms
2025-04-28T06:09:23.802949Z	warn	ca	ca request failed, starting attempt 2 in 215.975382ms
2025-04-28T06:09:24.019525Z	warn	ca	ca request failed, starting attempt 3 in 428.309685ms
2025-04-28T06:09:24.448223Z	warn	ca	ca request failed, starting attempt 4 in 777.286726ms
2025-04-28T06:09:25.226301Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:57677->10.96.0.10:53: i/o timeout"
2025-04-28T06:09:25.226869Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:57677->10.96.0.10:53: i/o timeout"
2025-04-28T06:09:25.226878Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:57677->10.96.0.10:53: i/o timeout"
2025-04-28T06:09:59.459067Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 995s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:10:25.753537Z	warn	ca	ca request failed, starting attempt 1 in 107.835865ms
2025-04-28T06:10:25.862026Z	warn	ca	ca request failed, starting attempt 2 in 208.605284ms
2025-04-28T06:10:26.071663Z	warn	ca	ca request failed, starting attempt 3 in 422.245947ms
2025-04-28T06:10:26.494427Z	warn	ca	ca request failed, starting attempt 4 in 798.361984ms
2025-04-28T06:10:27.293058Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:10:27.293738Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:10:27.293750Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:10:35.799154Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1032s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:11:19.758800Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1076s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:11:27.519023Z	warn	ca	ca request failed, starting attempt 1 in 98.970366ms
2025-04-28T06:11:27.618493Z	warn	ca	ca request failed, starting attempt 2 in 201.782891ms
2025-04-28T06:11:27.821068Z	warn	ca	ca request failed, starting attempt 3 in 438.648541ms
2025-04-28T06:11:28.015119Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1084s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:47453->10.96.0.10:53: i/o timeout"	thread=15
2025-04-28T06:11:28.260622Z	warn	ca	ca request failed, starting attempt 4 in 855.929542ms
2025-04-28T06:11:29.116752Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:47453->10.96.0.10:53: i/o timeout"
2025-04-28T06:11:29.117227Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:47453->10.96.0.10:53: i/o timeout"
2025-04-28T06:11:29.117236Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:47453->10.96.0.10:53: i/o timeout"
2025-04-28T06:12:03.646899Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1119s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:12:29.575017Z	warn	ca	ca request failed, starting attempt 1 in 102.146458ms
2025-04-28T06:12:29.677355Z	warn	ca	ca request failed, starting attempt 2 in 197.685045ms
2025-04-28T06:12:29.875995Z	warn	ca	ca request failed, starting attempt 3 in 387.820251ms
2025-04-28T06:12:30.264660Z	warn	ca	ca request failed, starting attempt 4 in 838.921711ms
2025-04-28T06:12:31.104578Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:12:31.105134Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:12:31.105143Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:12:53.142262Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1169s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:13:31.406135Z	warn	ca	ca request failed, starting attempt 1 in 93.229931ms
2025-04-28T06:13:31.499912Z	warn	ca	ca request failed, starting attempt 2 in 190.095868ms
2025-04-28T06:13:31.690176Z	warn	ca	ca request failed, starting attempt 3 in 432.723192ms
2025-04-28T06:13:32.123906Z	warn	ca	ca request failed, starting attempt 4 in 782.785983ms
2025-04-28T06:13:32.907758Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:13:32.908275Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:13:32.908295Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:13:40.816507Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1217s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:14:17.697035Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1254s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:14:33.223737Z	warn	ca	ca request failed, starting attempt 1 in 95.754553ms
2025-04-28T06:14:33.320150Z	warn	ca	ca request failed, starting attempt 2 in 193.797366ms
2025-04-28T06:14:33.514642Z	warn	ca	ca request failed, starting attempt 3 in 419.413121ms
2025-04-28T06:14:33.764050Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1270s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:53160->10.96.0.10:53: i/o timeout"	thread=15
2025-04-28T06:14:33.934333Z	warn	ca	ca request failed, starting attempt 4 in 863.467718ms
2025-04-28T06:14:34.798124Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:53160->10.96.0.10:53: i/o timeout"
2025-04-28T06:14:34.798653Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:53160->10.96.0.10:53: i/o timeout"
2025-04-28T06:14:34.798662Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:53160->10.96.0.10:53: i/o timeout"
2025-04-28T06:15:10.867835Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1307s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:15:35.090322Z	warn	ca	ca request failed, starting attempt 1 in 90.981352ms
2025-04-28T06:15:35.181805Z	warn	ca	ca request failed, starting attempt 2 in 197.34741ms
2025-04-28T06:15:35.380227Z	warn	ca	ca request failed, starting attempt 3 in 423.995483ms
2025-04-28T06:15:35.805015Z	warn	ca	ca request failed, starting attempt 4 in 782.105205ms
2025-04-28T06:15:36.587442Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:15:36.588064Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:15:36.588075Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:15:48.939179Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1345s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:16:24.839847Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1381s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:16:36.866198Z	warn	ca	ca request failed, starting attempt 1 in 90.761658ms
2025-04-28T06:16:36.957582Z	warn	ca	ca request failed, starting attempt 2 in 203.74764ms
2025-04-28T06:16:37.162057Z	warn	ca	ca request failed, starting attempt 3 in 395.925085ms
2025-04-28T06:16:37.558834Z	warn	ca	ca request failed, starting attempt 4 in 759.068584ms
2025-04-28T06:16:38.318190Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:32803->10.96.0.10:53: i/o timeout"
2025-04-28T06:16:38.318784Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:32803->10.96.0.10:53: i/o timeout"
2025-04-28T06:16:38.318794Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:32803->10.96.0.10:53: i/o timeout"
2025-04-28T06:17:09.190836Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1425s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:17:38.823388Z	warn	ca	ca request failed, starting attempt 1 in 101.359967ms
2025-04-28T06:17:38.925814Z	warn	ca	ca request failed, starting attempt 2 in 215.547287ms
2025-04-28T06:17:39.142373Z	warn	ca	ca request failed, starting attempt 3 in 399.781075ms
2025-04-28T06:17:39.543086Z	warn	ca	ca request failed, starting attempt 4 in 732.12153ms
2025-04-28T06:17:40.275687Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:17:40.276268Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:17:40.276278Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:17:40.399484Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1456s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:18:00.149257Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1476s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:43428->10.96.0.10:53: i/o timeout"	thread=15
2025-04-28T06:18:37.358645Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1513s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:18:40.605030Z	warn	ca	ca request failed, starting attempt 1 in 101.876458ms
2025-04-28T06:18:40.707424Z	warn	ca	ca request failed, starting attempt 2 in 182.326275ms
2025-04-28T06:18:40.889918Z	warn	ca	ca request failed, starting attempt 3 in 426.905405ms
2025-04-28T06:18:41.317696Z	warn	ca	ca request failed, starting attempt 4 in 811.870577ms
2025-04-28T06:18:42.129815Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:43847->10.96.0.10:53: i/o timeout"
2025-04-28T06:18:42.130384Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:43847->10.96.0.10:53: i/o timeout"
2025-04-28T06:18:42.130422Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:43847->10.96.0.10:53: i/o timeout"
2025-04-28T06:19:07.398902Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1543s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:19:42.342271Z	warn	ca	ca request failed, starting attempt 1 in 97.673132ms
2025-04-28T06:19:42.440608Z	warn	ca	ca request failed, starting attempt 2 in 213.19098ms
2025-04-28T06:19:42.654085Z	warn	ca	ca request failed, starting attempt 3 in 399.277923ms
2025-04-28T06:19:43.053655Z	warn	ca	ca request failed, starting attempt 4 in 807.248245ms
2025-04-28T06:19:43.861604Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:19:43.862183Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:19:43.862192Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:19:46.452954Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1582s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:20:03.691549Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1600s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:59650->10.96.0.10:53: i/o timeout"	thread=15
2025-04-28T06:20:44.286437Z	warn	ca	ca request failed, starting attempt 1 in 109.894696ms
2025-04-28T06:20:44.396861Z	warn	ca	ca request failed, starting attempt 2 in 217.708893ms
2025-04-28T06:20:44.615359Z	warn	ca	ca request failed, starting attempt 3 in 383.741342ms
2025-04-28T06:20:45.000053Z	warn	ca	ca request failed, starting attempt 4 in 829.885857ms
2025-04-28T06:20:45.830106Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:20:45.830830Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:20:45.830843Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:20:52.596860Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1648s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:21:33.621549Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1689s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:21:46.031651Z	warn	ca	ca request failed, starting attempt 1 in 102.542268ms
2025-04-28T06:21:46.135101Z	warn	ca	ca request failed, starting attempt 2 in 180.467522ms
2025-04-28T06:21:46.316605Z	warn	ca	ca request failed, starting attempt 3 in 401.037352ms
2025-04-28T06:21:46.718336Z	warn	ca	ca request failed, starting attempt 4 in 869.624329ms
2025-04-28T06:21:47.588536Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:57362->10.96.0.10:53: i/o timeout"
2025-04-28T06:21:47.589073Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:57362->10.96.0.10:53: i/o timeout"
2025-04-28T06:21:47.589084Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:57362->10.96.0.10:53: i/o timeout"
2025-04-28T06:22:14.394994Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1730s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:22:27.600551Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1743s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:57794->10.96.0.10:53: i/o timeout"	thread=15
2025-04-28T06:22:47.970697Z	warn	ca	ca request failed, starting attempt 1 in 103.50953ms
2025-04-28T06:22:48.075079Z	warn	ca	ca request failed, starting attempt 2 in 192.952121ms
2025-04-28T06:22:48.268574Z	warn	ca	ca request failed, starting attempt 3 in 437.615484ms
2025-04-28T06:22:48.707457Z	warn	ca	ca request failed, starting attempt 4 in 721.654837ms
2025-04-28T06:22:49.430118Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:22:49.430664Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:22:49.430674Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:23:00.782729Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1777s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:23:37.462566Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1813s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:23:49.836732Z	warn	ca	ca request failed, starting attempt 1 in 93.918943ms
2025-04-28T06:23:49.931172Z	warn	ca	ca request failed, starting attempt 2 in 196.247301ms
2025-04-28T06:23:49.993026Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1826s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:35806->10.96.0.10:53: i/o timeout"	thread=15
2025-04-28T06:23:50.128391Z	warn	ca	ca request failed, starting attempt 3 in 392.514375ms
2025-04-28T06:23:50.521107Z	warn	ca	ca request failed, starting attempt 4 in 733.418432ms
2025-04-28T06:23:51.255073Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:35806->10.96.0.10:53: i/o timeout"
2025-04-28T06:23:51.255695Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:35806->10.96.0.10:53: i/o timeout"
2025-04-28T06:23:51.255709Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:35806->10.96.0.10:53: i/o timeout"
2025-04-28T06:24:14.351144Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1850s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:24:31.270051Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1867s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:39490->10.96.0.10:53: i/o timeout"	thread=15
2025-04-28T06:24:51.602576Z	warn	ca	ca request failed, starting attempt 1 in 93.837302ms
2025-04-28T06:24:51.696948Z	warn	ca	ca request failed, starting attempt 2 in 195.7654ms
2025-04-28T06:24:51.893442Z	warn	ca	ca request failed, starting attempt 3 in 362.612162ms
2025-04-28T06:24:52.257231Z	warn	ca	ca request failed, starting attempt 4 in 785.869086ms
2025-04-28T06:24:53.044165Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:24:53.044756Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:24:53.044767Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:25:01.197644Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1897s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:25:40.728288Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1937s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:25:53.164676Z	warn	ca	ca request failed, starting attempt 1 in 107.396884ms
2025-04-28T06:25:53.273088Z	warn	ca	ca request failed, starting attempt 2 in 181.320737ms
2025-04-28T06:25:53.455652Z	warn	ca	ca request failed, starting attempt 3 in 379.933292ms
2025-04-28T06:25:53.836383Z	warn	ca	ca request failed, starting attempt 4 in 761.275139ms
2025-04-28T06:25:54.598567Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:44658->10.96.0.10:53: i/o timeout"
2025-04-28T06:25:54.599105Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:44658->10.96.0.10:53: i/o timeout"
2025-04-28T06:25:54.599114Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:44658->10.96.0.10:53: i/o timeout"
2025-04-28T06:26:21.358594Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1977s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:26:34.605258Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 1990s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:40401->10.96.0.10:53: i/o timeout"	thread=15
2025-04-28T06:26:55.388469Z	warn	ca	ca request failed, starting attempt 1 in 94.347475ms
2025-04-28T06:26:55.483796Z	warn	ca	ca request failed, starting attempt 2 in 201.171728ms
2025-04-28T06:26:55.685159Z	warn	ca	ca request failed, starting attempt 3 in 437.361786ms
2025-04-28T06:26:56.122863Z	warn	ca	ca request failed, starting attempt 4 in 818.490809ms
2025-04-28T06:26:56.942327Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:26:56.942859Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:26:56.942869Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:27:18.492164Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 2034s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:27:57.479088Z	warn	ca	ca request failed, starting attempt 1 in 95.583832ms
2025-04-28T06:27:57.575460Z	warn	ca	ca request failed, starting attempt 2 in 205.769491ms
2025-04-28T06:27:57.781953Z	warn	ca	ca request failed, starting attempt 3 in 416.800525ms
2025-04-28T06:27:58.199682Z	warn	ca	ca request failed, starting attempt 4 in 780.576097ms
2025-04-28T06:27:58.980606Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:27:58.981197Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:27:58.981207Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:28:00.469707Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 2076s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:28:49.687719Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 2126s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:28:59.108759Z	warn	ca	ca request failed, starting attempt 1 in 104.09732ms
2025-04-28T06:28:59.213101Z	warn	ca	ca request failed, starting attempt 2 in 181.85497ms
2025-04-28T06:28:59.395601Z	warn	ca	ca request failed, starting attempt 3 in 413.898905ms
2025-04-28T06:28:59.810460Z	warn	ca	ca request failed, starting attempt 4 in 792.644525ms
2025-04-28T06:29:00.603496Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:46108->10.96.0.10:53: i/o timeout"
2025-04-28T06:29:00.604016Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:46108->10.96.0.10:53: i/o timeout"
2025-04-28T06:29:00.604025Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:46108->10.96.0.10:53: i/o timeout"
2025-04-28T06:29:27.682904Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 2164s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:30:00.795184Z	warn	ca	ca request failed, starting attempt 1 in 102.913423ms
2025-04-28T06:30:00.898534Z	warn	ca	ca request failed, starting attempt 2 in 213.53525ms
2025-04-28T06:30:01.112974Z	warn	ca	ca request failed, starting attempt 3 in 392.891469ms
2025-04-28T06:30:01.506760Z	warn	ca	ca request failed, starting attempt 4 in 749.100846ms
2025-04-28T06:30:02.256804Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:39553->10.96.0.10:53: i/o timeout"
2025-04-28T06:30:02.257382Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:39553->10.96.0.10:53: i/o timeout"
2025-04-28T06:30:02.257395Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:39553->10.96.0.10:53: i/o timeout"
2025-04-28T06:30:07.142973Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 2203s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:30:22.490583Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 2218s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:56020->10.96.0.10:53: i/o timeout"	thread=15
2025-04-28T06:31:01.926992Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 2258s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:31:02.644260Z	warn	ca	ca request failed, starting attempt 1 in 101.219915ms
2025-04-28T06:31:02.745652Z	warn	ca	ca request failed, starting attempt 2 in 199.994126ms
2025-04-28T06:31:02.946277Z	warn	ca	ca request failed, starting attempt 3 in 408.450317ms
2025-04-28T06:31:03.274303Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 2259s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:39144->10.96.0.10:53: i/o timeout"	thread=15
2025-04-28T06:31:03.355678Z	warn	ca	ca request failed, starting attempt 4 in 747.861389ms
2025-04-28T06:31:04.104837Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:39144->10.96.0.10:53: i/o timeout"
2025-04-28T06:31:04.105510Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:39144->10.96.0.10:53: i/o timeout"
2025-04-28T06:31:04.105520Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:39144->10.96.0.10:53: i/o timeout"
2025-04-28T06:31:52.660420Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 2308s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:32:04.266247Z	warn	ca	ca request failed, starting attempt 1 in 93.258351ms
2025-04-28T06:32:04.360629Z	warn	ca	ca request failed, starting attempt 2 in 200.906907ms
2025-04-28T06:32:04.562188Z	warn	ca	ca request failed, starting attempt 3 in 430.023631ms
2025-04-28T06:32:04.993052Z	warn	ca	ca request failed, starting attempt 4 in 824.362946ms
2025-04-28T06:32:05.818107Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:55547->10.96.0.10:53: i/o timeout"
2025-04-28T06:32:05.818655Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:55547->10.96.0.10:53: i/o timeout"
2025-04-28T06:32:05.818664Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:55547->10.96.0.10:53: i/o timeout"
2025-04-28T06:32:28.846058Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 2345s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:32:45.829204Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 2362s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:59067->10.96.0.10:53: i/o timeout"	thread=15
2025-04-28T06:33:05.930460Z	warn	ca	ca request failed, starting attempt 1 in 101.660512ms
2025-04-28T06:33:06.032838Z	warn	ca	ca request failed, starting attempt 2 in 207.999414ms
2025-04-28T06:33:06.241456Z	warn	ca	ca request failed, starting attempt 3 in 376.641308ms
2025-04-28T06:33:06.619171Z	warn	ca	ca request failed, starting attempt 4 in 798.126293ms
2025-04-28T06:33:07.417493Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:33:07.417984Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:33:07.417994Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:33:34.983732Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 2411s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:33:47.427220Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 2423s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:54312->10.96.0.10:53: i/o timeout"	thread=15
2025-04-28T06:34:07.917167Z	warn	ca	ca request failed, starting attempt 1 in 106.360933ms
2025-04-28T06:34:08.024699Z	warn	ca	ca request failed, starting attempt 2 in 209.08956ms
2025-04-28T06:34:08.234183Z	warn	ca	ca request failed, starting attempt 3 in 380.461119ms
2025-04-28T06:34:08.615924Z	warn	ca	ca request failed, starting attempt 4 in 864.381791ms
2025-04-28T06:34:09.481418Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:34:09.481937Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:34:09.481963Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:34:17.272804Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 2453s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:34:29.047278Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 2465s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:52086->10.96.0.10:53: i/o timeout"	thread=15
2025-04-28T06:34:52.245992Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 2488s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:35:09.735091Z	warn	ca	ca request failed, starting attempt 1 in 97.774754ms
2025-04-28T06:35:09.833520Z	warn	ca	ca request failed, starting attempt 2 in 189.491203ms
2025-04-28T06:35:10.024105Z	warn	ca	ca request failed, starting attempt 3 in 404.862612ms
2025-04-28T06:35:10.055921Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 2506s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:59179->10.96.0.10:53: i/o timeout"	thread=15
2025-04-28T06:35:10.429392Z	warn	ca	ca request failed, starting attempt 4 in 806.372042ms
2025-04-28T06:35:11.236523Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:59179->10.96.0.10:53: i/o timeout"
2025-04-28T06:35:11.237124Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:59179->10.96.0.10:53: i/o timeout"
2025-04-28T06:35:11.237135Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:59179->10.96.0.10:53: i/o timeout"
2025-04-28T06:35:55.020893Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 2551s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:36:11.482723Z	warn	ca	ca request failed, starting attempt 1 in 101.486229ms
2025-04-28T06:36:11.585314Z	warn	ca	ca request failed, starting attempt 2 in 196.734311ms
2025-04-28T06:36:11.782865Z	warn	ca	ca request failed, starting attempt 3 in 385.896029ms
2025-04-28T06:36:12.169586Z	warn	ca	ca request failed, starting attempt 4 in 777.764042ms
2025-04-28T06:36:12.948141Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:38895->10.96.0.10:53: i/o timeout"
2025-04-28T06:36:12.948605Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:38895->10.96.0.10:53: i/o timeout"
2025-04-28T06:36:12.948614Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:38895->10.96.0.10:53: i/o timeout"
2025-04-28T06:36:44.838553Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 2601s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:37:13.134793Z	warn	ca	ca request failed, starting attempt 1 in 101.491555ms
2025-04-28T06:37:13.237102Z	warn	ca	ca request failed, starting attempt 2 in 208.748544ms
2025-04-28T06:37:13.446665Z	warn	ca	ca request failed, starting attempt 3 in 401.758116ms
2025-04-28T06:37:13.849461Z	warn	ca	ca request failed, starting attempt 4 in 872.089265ms
2025-04-28T06:37:14.722345Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:37:14.722760Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:37:14.722769Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:37:21.524337Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 2637s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:38:11.499716Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 2687s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:38:15.135201Z	warn	ca	ca request failed, starting attempt 1 in 99.3292ms
2025-04-28T06:38:15.235544Z	warn	ca	ca request failed, starting attempt 2 in 218.701413ms
2025-04-28T06:38:15.455051Z	warn	ca	ca request failed, starting attempt 3 in 397.977343ms
2025-04-28T06:38:15.853828Z	warn	ca	ca request failed, starting attempt 4 in 757.726693ms
2025-04-28T06:38:16.611925Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:44515->10.96.0.10:53: i/o timeout"
2025-04-28T06:38:16.612433Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:44515->10.96.0.10:53: i/o timeout"
2025-04-28T06:38:16.612445Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.235:44515->10.96.0.10:53: i/o timeout"
2025-04-28T06:38:50.050089Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 2726s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:39:16.998873Z	warn	ca	ca request failed, starting attempt 1 in 103.182552ms
2025-04-28T06:39:17.103249Z	warn	ca	ca request failed, starting attempt 2 in 196.855658ms
2025-04-28T06:39:17.300689Z	warn	ca	ca request failed, starting attempt 3 in 438.238347ms
2025-04-28T06:39:17.739228Z	warn	ca	ca request failed, starting attempt 4 in 821.07425ms
2025-04-28T06:39:18.561043Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:39:18.561613Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:39:18.561623Z	warn	sds	failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"
2025-04-28T06:39:28.561349Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 2764s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:40:06.483013Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:193	DeltaAggregatedResources gRPC config stream to xds-grpc closed since 2802s ago: 14, connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout"	thread=15
2025-04-28T06:40:09.748255Z	warn	ca	ca request failed, starting attempt 1 in 97.496323ms
2025-04-28T06:40:09.846601Z	warn	ca	ca request failed, starting attempt 2 in 183.085117ms
2025-04-28T06:40:10.029930Z	warn	ca	ca request failed, starting attempt 3 in 438.783122ms
2025-04-28T06:40:10.469559Z	warn	ca	ca request failed, starting attempt 4 in 875.642593ms
2025-04-28T06:40:11.345630Z	error	citadelclient	failed to sign CSR: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: no such host"
2025-04-28T06:40:11.346163Z	error	cache	resource:default failed to sign: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: no such host"
2025-04-28T06:40:11.346667Z	warning	envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:155	StreamSecrets gRPC config stream to sds-grpc closed: 2, failed to generate secret for default: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: no such host"	thread=15
[2025-04-28T06:56:46.289Z] "- - HTTP/1.1" 400 DPE http1.codec_error - "-" 0 11 0 - "-" "-" "-" "-" "-" - - 10.244.1.235:8080 10.244.2.0:20109 - -
[2025-04-28T06:56:46.289Z] "- - HTTP/1.1" 400 DPE http1.codec_error - "-" 0 11 0 - "-" "-" "-" "-" "-" - - 10.244.1.235:8080 10.244.2.0:20946 - -
[2025-04-28T06:56:46.299Z] "- - HTTP/1.1" 400 DPE http1.codec_error - "-" 0 11 0 - "-" "-" "-" "-" "-" - - 10.244.1.235:8080 10.244.2.0:60547 - -
[2025-04-28T06:56:46.300Z] "- - HTTP/1.1" 400 DPE http1.codec_error - "-" 0 11 0 - "-" "-" "-" "-" "-" - - 10.244.1.235:8080 10.244.2.0:22926 - -
[2025-04-28T06:57:35.138Z] "- - HTTP/1.1" 400 DPE http1.codec_error - "-" 0 11 0 - "-" "-" "-" "-" "-" - - 10.244.1.235:8080 10.244.0.0:40237 - -
[2025-04-28T06:57:35.138Z] "- - HTTP/1.1" 400 DPE http1.codec_error - "-" 0 11 0 - "-" "-" "-" "-" "-" - - 10.244.1.235:8080 10.244.0.0:12857 - -
[2025-04-28T06:57:35.151Z] "- - HTTP/1.1" 400 DPE http1.codec_error - "-" 0 11 0 - "-" "-" "-" "-" "-" - - 10.244.1.235:8080 10.244.0.0:5128 - -
[2025-04-28T06:57:35.151Z] "- - HTTP/1.1" 400 DPE http1.codec_error - "-" 0 11 0 - "-" "-" "-" "-" "-" - - 10.244.1.235:8080 10.244.0.0:43903 - -

Pod: activator-54c999bcbc-4vglt (Namespace: knative-serving)

Logs (filtered)

{"severity":"ERROR","timestamp":"2025-04-28T06:43:45.407803125Z","logger":"activator","caller":"websocket/connection.go:141","message":"Websocket connection could not be established","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"websocket: bad handshake","request":"HTTP/1.1 503 Service Unavailable\r\nConnection: close\r\nContent-Length: 19\r\nContent-Type: text/plain\r\nDate: Mon, 28 Apr 2025 06:43:45 GMT\r\nServer: envoy\r\n\r\nno healthy upstream","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func1\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:141\nknative.dev/pkg/websocket.NewDurableConnection.func2.(*ManagedConnection).connect.func1\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:225\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\tk8s.io/apimachinery@v0.27.6/pkg/util/wait/wait.go:145\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\tk8s.io/apimachinery@v0.27.6/pkg/util/wait/backoff.go:461\nknative.dev/pkg/websocket.(*ManagedConnection).connect\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:222\nknative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:162"}
{"severity":"ERROR","timestamp":"2025-04-28T06:43:45.530471686Z","logger":"activator","caller":"websocket/connection.go:141","message":"Websocket connection could not be established","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"websocket: bad handshake","request":"HTTP/1.1 503 Service Unavailable\r\nConnection: close\r\nContent-Length: 19\r\nContent-Type: text/plain\r\nDate: Mon, 28 Apr 2025 06:43:45 GMT\r\nServer: envoy\r\n\r\nno healthy upstream","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func1\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:141\nknative.dev/pkg/websocket.NewDurableConnection.func2.(*ManagedConnection).connect.func1\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:225\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\tk8s.io/apimachinery@v0.27.6/pkg/util/wait/wait.go:145\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\tk8s.io/apimachinery@v0.27.6/pkg/util/wait/backoff.go:461\nknative.dev/pkg/websocket.(*ManagedConnection).connect\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:222\nknative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:162"}
{"severity":"ERROR","timestamp":"2025-04-28T06:43:45.700364666Z","logger":"activator","caller":"websocket/connection.go:141","message":"Websocket connection could not be established","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"websocket: bad handshake","request":"HTTP/1.1 503 Service Unavailable\r\nConnection: close\r\nContent-Length: 19\r\nContent-Type: text/plain\r\nDate: Mon, 28 Apr 2025 06:43:45 GMT\r\nServer: envoy\r\n\r\nno healthy upstream","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func1\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:141\nknative.dev/pkg/websocket.NewDurableConnection.func2.(*ManagedConnection).connect.func1\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:225\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\tk8s.io/apimachinery@v0.27.6/pkg/util/wait/wait.go:145\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\tk8s.io/apimachinery@v0.27.6/pkg/util/wait/backoff.go:461\nknative.dev/pkg/websocket.(*ManagedConnection).connect\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:222\nknative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:162"}
{"severity":"ERROR","timestamp":"2025-04-28T06:43:45.892712991Z","logger":"activator","caller":"websocket/connection.go:141","message":"Websocket connection could not be established","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"websocket: bad handshake","request":"HTTP/1.1 503 Service Unavailable\r\nConnection: close\r\nContent-Length: 19\r\nContent-Type: text/plain\r\nDate: Mon, 28 Apr 2025 06:43:45 GMT\r\nServer: envoy\r\n\r\nno healthy upstream","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func1\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:141\nknative.dev/pkg/websocket.NewDurableConnection.func2.(*ManagedConnection).connect.func1\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:225\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\tk8s.io/apimachinery@v0.27.6/pkg/util/wait/wait.go:145\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\tk8s.io/apimachinery@v0.27.6/pkg/util/wait/backoff.go:461\nknative.dev/pkg/websocket.(*ManagedConnection).connect\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:222\nknative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:162"}
{"severity":"WARNING","timestamp":"2025-04-28T06:43:46.075655807Z","logger":"activator","caller":"handler/healthz_handler.go:36","message":"Healthcheck failed: connection has not yet been established","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt"}
{"severity":"ERROR","timestamp":"2025-04-28T06:43:46.168683609Z","logger":"activator","caller":"websocket/connection.go:141","message":"Websocket connection could not be established","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"websocket: bad handshake","request":"HTTP/1.1 503 Service Unavailable\r\nConnection: close\r\nContent-Length: 19\r\nContent-Type: text/plain\r\nDate: Mon, 28 Apr 2025 06:43:45 GMT\r\nServer: envoy\r\n\r\nno healthy upstream","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func1\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:141\nknative.dev/pkg/websocket.NewDurableConnection.func2.(*ManagedConnection).connect.func1\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:225\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\tk8s.io/apimachinery@v0.27.6/pkg/util/wait/wait.go:145\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\tk8s.io/apimachinery@v0.27.6/pkg/util/wait/backoff.go:461\nknative.dev/pkg/websocket.(*ManagedConnection).connect\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:222\nknative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:162"}
{"severity":"ERROR","timestamp":"2025-04-28T06:45:41.879338063Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"websocket: close 1006 (abnormal closure): unexpected EOF","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-04-28T12:45:42.045654866Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"websocket: close 1006 (abnormal closure): unexpected EOF","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-04-28T18:45:42.013433841Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"websocket: close 1006 (abnormal closure): unexpected EOF","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-04-29T00:45:42.036218556Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"websocket: close 1006 (abnormal closure): unexpected EOF","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-04-29T06:45:42.15831237Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"read tcp 10.244.1.242:54724->10.96.80.146:8080: read: connection reset by peer","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-04-29T12:45:42.199025584Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"read tcp 10.244.1.242:39686->10.96.80.146:8080: read: connection reset by peer","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-04-29T18:45:42.23296617Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"read tcp 10.244.1.242:42802->10.96.80.146:8080: read: connection reset by peer","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-04-30T00:45:42.35703574Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"read tcp 10.244.1.242:50874->10.96.80.146:8080: read: connection reset by peer","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-04-30T06:45:42.391478831Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"read tcp 10.244.1.242:40658->10.96.80.146:8080: read: connection reset by peer","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-04-30T12:45:42.328240166Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"read tcp 10.244.1.242:55784->10.96.80.146:8080: read: connection reset by peer","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-04-30T18:45:42.366920196Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"read tcp 10.244.1.242:45682->10.96.80.146:8080: read: connection reset by peer","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-05-01T00:45:42.457422759Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"read tcp 10.244.1.242:41232->10.96.80.146:8080: read: connection reset by peer","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-05-01T06:45:42.460530234Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"read tcp 10.244.1.242:33282->10.96.80.146:8080: read: connection reset by peer","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-05-01T12:45:42.47684161Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"read tcp 10.244.1.242:33694->10.96.80.146:8080: read: connection reset by peer","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-05-01T18:45:42.713067733Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"read tcp 10.244.1.242:47610->10.96.80.146:8080: read: connection reset by peer","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-05-02T00:45:42.618579165Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"read tcp 10.244.1.242:46738->10.96.80.146:8080: read: connection reset by peer","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-05-02T06:45:42.683572298Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"read tcp 10.244.1.242:60724->10.96.80.146:8080: read: connection reset by peer","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-05-02T12:45:42.734917403Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"read tcp 10.244.1.242:36622->10.96.80.146:8080: read: connection reset by peer","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-05-02T18:45:42.718647041Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"read tcp 10.244.1.242:44680->10.96.80.146:8080: read: connection reset by peer","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-05-03T00:45:42.898145871Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"read tcp 10.244.1.242:39028->10.96.80.146:8080: read: connection reset by peer","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-05-03T06:45:42.779084919Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"read tcp 10.244.1.242:57272->10.96.80.146:8080: read: connection reset by peer","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-05-03T12:45:42.89156129Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"read tcp 10.244.1.242:39558->10.96.80.146:8080: read: connection reset by peer","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-05-03T18:45:42.953855487Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"read tcp 10.244.1.242:46902->10.96.80.146:8080: read: connection reset by peer","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-05-04T00:45:42.880771095Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"read tcp 10.244.1.242:58690->10.96.80.146:8080: read: connection reset by peer","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-05-04T06:45:42.916194684Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"read tcp 10.244.1.242:53024->10.96.80.146:8080: read: connection reset by peer","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-05-04T12:45:43.095250442Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"websocket: close 1006 (abnormal closure): unexpected EOF","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-05-04T18:45:43.110597217Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"websocket: close 1006 (abnormal closure): unexpected EOF","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-05-05T00:45:43.025207331Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"read tcp 10.244.1.242:53296->10.96.80.146:8080: read: connection reset by peer","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-05-05T06:45:43.106370326Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"websocket: close 1006 (abnormal closure): unexpected EOF","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-05-05T12:45:44.602457218Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"websocket: close 1006 (abnormal closure): unexpected EOF","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-05-05T18:45:46.284586977Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"read tcp 10.244.1.242:58000->10.96.80.146:8080: read: connection reset by peer","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-05-06T00:45:47.944211898Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"websocket: close 1006 (abnormal closure): unexpected EOF","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-05-06T06:45:49.450912866Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"read tcp 10.244.1.242:51160->10.96.80.146:8080: read: connection reset by peer","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-05-06T12:44:51.130237435Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"websocket: close 1006 (abnormal closure): unexpected EOF","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-05-06T18:44:52.806207476Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"read tcp 10.244.1.242:48086->10.96.80.146:8080: read: connection reset by peer","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-05-07T00:44:54.500627494Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"websocket: close 1006 (abnormal closure): unexpected EOF","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-05-07T06:44:56.145729999Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"read tcp 10.244.1.242:58356->10.96.80.146:8080: read: connection reset by peer","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-05-07T12:44:57.730982026Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"websocket: close 1006 (abnormal closure): unexpected EOF","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-05-07T18:44:59.329128192Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"read tcp 10.244.1.242:35236->10.96.80.146:8080: read: connection reset by peer","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}
{"severity":"ERROR","timestamp":"2025-05-08T00:45:01.098673094Z","logger":"activator","caller":"websocket/connection.go:168","message":"Connection to ws://autoscaler.knative-serving.svc.cluster.local:8080 broke down, reconnecting...","commit":"9d21588","knative.dev/controller":"activator","knative.dev/pod":"activator-54c999bcbc-4vglt","error":"websocket: close 1006 (abnormal closure): unexpected EOF","stacktrace":"knative.dev/pkg/websocket.NewDurableConnection.func2\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/websocket/connection.go:168"}

Pod: controller-854dcfd65b-krjgp (Namespace: knative-serving)

Logs (filtered)

{"severity":"ERROR","timestamp":"2025-04-28T06:43:46.889603183Z","logger":"controller","caller":"reconciler/reconciler.go:280","message":"Returned an error","commit":"9d21588","knative.dev/pod":"controller-854dcfd65b-krjgp","knative.dev/controller":"knative.dev.networking.pkg.certificates.reconciler.reconciler","knative.dev/kind":"core.Secret","knative.dev/traceid":"7893f51f-b9ad-43b6-84c3-c157cc2605f3","knative.dev/key":"knative-serving/knative-serving-certs","targetMethod":"ReconcileKind","error":"Operation cannot be fulfilled on secrets \"serving-certs-ctrl-ca\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/networking/pkg/certificates/reconciler.(*reconcilerImpl).Reconcile\n\tknative.dev/networking@v0.0.0-20231017124814-2a7676e912b7/pkg/certificates/reconciler/reconciler.go:280\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:542\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:491"}
{"severity":"ERROR","timestamp":"2025-04-28T06:43:46.889695755Z","logger":"controller","caller":"controller/controller.go:566","message":"Reconcile error","commit":"9d21588","knative.dev/pod":"controller-854dcfd65b-krjgp","knative.dev/controller":"knative.dev.networking.pkg.certificates.reconciler.reconciler","knative.dev/kind":"core.Secret","knative.dev/traceid":"7893f51f-b9ad-43b6-84c3-c157cc2605f3","knative.dev/key":"knative-serving/knative-serving-certs","duration":"973.741199ms","error":"Operation cannot be fulfilled on secrets \"serving-certs-ctrl-ca\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:491"}

Pod: net-istio-webhook-ff94788ff-tpprf (Namespace: knative-serving)

Logs (filtered)

{"severity":"ERROR","timestamp":"2025-04-28T06:43:46.055966191Z","logger":"net-istio-webhook.DefaultingWebhook","caller":"controller/controller.go:566","message":"Reconcile error","commit":"916ac51-dirty","knative.dev/traceid":"64667191-6185-4573-8148-d64120b741a3","knative.dev/key":"webhook.istio.networking.internal.knative.dev","duration":"21.795µs","error":"secret \"net-istio-webhook-certs\" is missing \"ca-cert.pem\" key","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:491"}
{"severity":"ERROR","timestamp":"2025-04-28T06:43:46.056139811Z","logger":"net-istio-webhook.ConfigMapWebhook","caller":"controller/controller.go:566","message":"Reconcile error","commit":"916ac51-dirty","knative.dev/traceid":"57d391cb-44f1-4590-8854-cf50e16d2986","knative.dev/key":"config.webhook.istio.networking.internal.knative.dev","duration":"17.524µs","error":"secret \"net-istio-webhook-certs\" is missing \"ca-cert.pem\" key","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:491"}
{"severity":"ERROR","timestamp":"2025-04-28T06:43:46.109091056Z","logger":"net-istio-webhook.ConfigMapWebhook","caller":"controller/controller.go:566","message":"Reconcile error","commit":"916ac51-dirty","knative.dev/traceid":"c9dd2221-b909-4f04-80dd-5cf27c64eb89","knative.dev/key":"config.webhook.istio.networking.internal.knative.dev","duration":"47.088851ms","error":"failed to update webhook: Operation cannot be fulfilled on validatingwebhookconfigurations.admissionregistration.k8s.io \"config.webhook.istio.networking.internal.knative.dev\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:491"}
{"severity":"ERROR","timestamp":"2025-04-28T06:43:46.109662177Z","logger":"net-istio-webhook.DefaultingWebhook","caller":"controller/controller.go:566","message":"Reconcile error","commit":"916ac51-dirty","knative.dev/traceid":"a2e7648b-2071-463c-a6ea-c391160c510c","knative.dev/key":"webhook.istio.networking.internal.knative.dev","duration":"44.649004ms","error":"failed to update webhook: Operation cannot be fulfilled on mutatingwebhookconfigurations.admissionregistration.k8s.io \"webhook.istio.networking.internal.knative.dev\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:491"}
{"severity":"ERROR","timestamp":"2025-05-04T12:43:46.310938126Z","logger":"net-istio-webhook.DefaultingWebhook","caller":"controller/controller.go:566","message":"Reconcile error","commit":"916ac51-dirty","knative.dev/traceid":"60eadb95-8143-453e-9e54-631844d7f7f4","knative.dev/key":"webhook.istio.networking.internal.knative.dev","duration":"286.069301ms","error":"failed to update webhook: Operation cannot be fulfilled on mutatingwebhookconfigurations.admissionregistration.k8s.io \"webhook.istio.networking.internal.knative.dev\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:491"}

Pod: webhook-56bccb4b4f-67mg6 (Namespace: knative-serving)

Logs (filtered)

{"severity":"ERROR","timestamp":"2025-04-28T06:43:46.542567994Z","logger":"webhook.ValidationWebhook","caller":"controller/controller.go:566","message":"Reconcile error","commit":"9d21588","knative.dev/pod":"webhook-56bccb4b4f-67mg6","knative.dev/traceid":"ce7b47ff-b988-42d0-be3c-76804d0344c0","knative.dev/key":"validation.webhook.serving.knative.dev","duration":"39.112µs","error":"secret \"webhook-certs\" is missing \"ca-cert.pem\" key","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:491"}
{"severity":"ERROR","timestamp":"2025-04-28T06:43:46.54278926Z","logger":"webhook.DefaultingWebhook","caller":"controller/controller.go:566","message":"Reconcile error","commit":"9d21588","knative.dev/pod":"webhook-56bccb4b4f-67mg6","knative.dev/traceid":"3c16c2f8-cd8d-4ba7-8f9e-3a3bf2d2e46b","knative.dev/key":"webhook.serving.knative.dev","duration":"17.099µs","error":"secret \"webhook-certs\" is missing \"ca-cert.pem\" key","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:491"}
{"severity":"ERROR","timestamp":"2025-04-28T06:43:46.54298686Z","logger":"webhook.ConfigMapWebhook","caller":"controller/controller.go:566","message":"Reconcile error","commit":"9d21588","knative.dev/pod":"webhook-56bccb4b4f-67mg6","knative.dev/traceid":"8dd2f27d-7153-4654-9a6a-832ff8660416","knative.dev/key":"config.webhook.serving.knative.dev","duration":"19.735µs","error":"secret \"webhook-certs\" is missing \"ca-cert.pem\" key","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:491"}
{"severity":"ERROR","timestamp":"2025-04-28T06:43:46.548755013Z","logger":"webhook.ConfigMapWebhook","caller":"controller/controller.go:566","message":"Reconcile error","commit":"9d21588","knative.dev/pod":"webhook-56bccb4b4f-67mg6","knative.dev/traceid":"9b1554b5-a8fd-4914-b218-fb36dbe2502c","knative.dev/key":"config.webhook.serving.knative.dev","duration":"20.216µs","error":"secret \"webhook-certs\" is missing \"ca-cert.pem\" key","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:491"}
{"severity":"ERROR","timestamp":"2025-04-28T06:43:46.548861054Z","logger":"webhook.ValidationWebhook","caller":"controller/controller.go:566","message":"Reconcile error","commit":"9d21588","knative.dev/pod":"webhook-56bccb4b4f-67mg6","knative.dev/traceid":"f2150e03-8cf5-4649-8403-163e2d50e3c0","knative.dev/key":"validation.webhook.serving.knative.dev","duration":"15.561µs","error":"secret \"webhook-certs\" is missing \"ca-cert.pem\" key","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:491"}
{"severity":"ERROR","timestamp":"2025-04-28T06:43:46.549027881Z","logger":"webhook.DefaultingWebhook","caller":"controller/controller.go:566","message":"Reconcile error","commit":"9d21588","knative.dev/pod":"webhook-56bccb4b4f-67mg6","knative.dev/traceid":"ee6e01f7-ada7-4c5d-9bb6-6caff00c9bea","knative.dev/key":"webhook.serving.knative.dev","duration":"16.321µs","error":"secret \"webhook-certs\" is missing \"ca-cert.pem\" key","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:491"}
{"severity":"ERROR","timestamp":"2025-04-28T06:43:46.592489349Z","logger":"webhook.DefaultingWebhook","caller":"controller/controller.go:566","message":"Reconcile error","commit":"9d21588","knative.dev/pod":"webhook-56bccb4b4f-67mg6","knative.dev/traceid":"31b481db-aba7-4c20-9330-9e222a437670","knative.dev/key":"webhook.serving.knative.dev","duration":"32.420573ms","error":"failed to update webhook: Operation cannot be fulfilled on mutatingwebhookconfigurations.admissionregistration.k8s.io \"webhook.serving.knative.dev\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:491"}
{"severity":"ERROR","timestamp":"2025-04-28T06:43:46.593002794Z","logger":"webhook.ValidationWebhook","caller":"controller/controller.go:566","message":"Reconcile error","commit":"9d21588","knative.dev/pod":"webhook-56bccb4b4f-67mg6","knative.dev/traceid":"9ae080ef-f2bb-492a-9207-d426dd6995f8","knative.dev/key":"validation.webhook.serving.knative.dev","duration":"32.442371ms","error":"failed to update webhook: Operation cannot be fulfilled on validatingwebhookconfigurations.admissionregistration.k8s.io \"validation.webhook.serving.knative.dev\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:491"}
{"severity":"ERROR","timestamp":"2025-04-28T06:43:46.593451992Z","logger":"webhook.ConfigMapWebhook","caller":"controller/controller.go:566","message":"Reconcile error","commit":"9d21588","knative.dev/pod":"webhook-56bccb4b4f-67mg6","knative.dev/traceid":"1d12fb99-d4fd-4381-97b7-d1e6508b5491","knative.dev/key":"config.webhook.serving.knative.dev","duration":"33.082878ms","error":"failed to update webhook: Operation cannot be fulfilled on validatingwebhookconfigurations.admissionregistration.k8s.io \"config.webhook.serving.knative.dev\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:491"}

Pod: kube-flannel-ds-brk8q (Namespace: kube-flannel)

Logs (filtered)

W0425 06:37:37.487615       1 reflector.go:462] pkg/subnet/kube/kube.go:470: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0425 06:38:43.365286       1 reflector.go:539] pkg/subnet/kube/kube.go:470: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90263": net/http: TLS handshake timeout
E0425 06:38:56.368289       1 reflector.go:147] pkg/subnet/kube/kube.go:470: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90263": net/http: TLS handshake timeout
W0425 06:40:34.672461       1 reflector.go:539] pkg/subnet/kube/kube.go:470: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90263": net/http: TLS handshake timeout
E0425 06:40:59.760522       1 reflector.go:147] pkg/subnet/kube/kube.go:470: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90263": net/http: TLS handshake timeout

Pod: kube-flannel-ds-clk96 (Namespace: kube-flannel)

Logs (filtered)

W0425 06:37:26.048012       1 reflector.go:462] pkg/subnet/kube/kube.go:470: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0425 06:37:37.560833       1 reflector.go:539] pkg/subnet/kube/kube.go:470: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90147": net/http: TLS handshake timeout
E0425 06:37:37.560937       1 reflector.go:147] pkg/subnet/kube/kube.go:470: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90147": net/http: TLS handshake timeout
W0425 06:37:49.913662       1 reflector.go:539] pkg/subnet/kube/kube.go:470: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90147": net/http: TLS handshake timeout
E0425 06:37:49.913833       1 reflector.go:147] pkg/subnet/kube/kube.go:470: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90147": net/http: TLS handshake timeout
W0425 06:38:05.787346       1 reflector.go:539] pkg/subnet/kube/kube.go:470: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90147": net/http: TLS handshake timeout
E0425 06:38:05.787433       1 reflector.go:147] pkg/subnet/kube/kube.go:470: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90147": net/http: TLS handshake timeout
W0425 06:38:27.343507       1 reflector.go:539] pkg/subnet/kube/kube.go:470: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90147": net/http: TLS handshake timeout
E0425 06:38:27.343559       1 reflector.go:147] pkg/subnet/kube/kube.go:470: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90147": net/http: TLS handshake timeout
W0425 06:38:53.784725       1 reflector.go:539] pkg/subnet/kube/kube.go:470: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90147": net/http: TLS handshake timeout
E0425 06:38:53.784783       1 reflector.go:147] pkg/subnet/kube/kube.go:470: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90147": net/http: TLS handshake timeout
W0425 06:39:31.634961       1 reflector.go:539] pkg/subnet/kube/kube.go:470: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90147": net/http: TLS handshake timeout
E0425 06:39:31.635030       1 reflector.go:147] pkg/subnet/kube/kube.go:470: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90147": net/http: TLS handshake timeout
W0425 06:40:39.136050       1 reflector.go:539] pkg/subnet/kube/kube.go:470: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90147": net/http: TLS handshake timeout
E0425 06:40:39.136143       1 reflector.go:147] pkg/subnet/kube/kube.go:470: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90147": net/http: TLS handshake timeout
W0425 06:41:25.254518       1 reflector.go:539] pkg/subnet/kube/kube.go:470: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90147": net/http: TLS handshake timeout
E0425 06:41:25.254616       1 reflector.go:147] pkg/subnet/kube/kube.go:470: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90147": net/http: TLS handshake timeout
W0425 06:42:18.199041       1 reflector.go:539] pkg/subnet/kube/kube.go:470: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90147": net/http: TLS handshake timeout
E0425 06:42:18.199150       1 reflector.go:147] pkg/subnet/kube/kube.go:470: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90147": net/http: TLS handshake timeout
E0425 07:10:12.759288       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:19:27.762139       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:26:33.764962       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:34:13.770547       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:42:55.773863       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:50:33.776420       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:56:02.779010       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:05:50.782190       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:14:33.785833       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:23:57.789012       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:32:28.791944       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:41:35.794718       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:49:14.797552       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:54:57.799796       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:03:08.801952       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:11:49.803987       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:20:17.806700       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:28:38.809536       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:36:12.811933       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:41:31.814225       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:49:28.816844       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:57:12.819808       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:06:25.822046       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:15:52.824626       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:21:56.827137       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:27:37.830604       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:36:13.833446       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:44:07.836374       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:53:07.839326       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:02:01.842369       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:09:49.845251       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:18:48.847927       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:26:31.850700       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:35:07.853604       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:42:36.855490       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:49:03.857874       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:58:06.859980       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:06:37.863226       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:15:22.865603       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:21:43.868063       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:26:57.870361       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:32:22.873085       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:38:02.875590       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:44:12.877991       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:51:32.880219       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:59:20.882971       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:06:24.885776       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:13:14.888072       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:19:25.891210       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:26:30.893908       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:35:04.896462       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:40:39.899331       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:49:35.902146       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:58:13.905257       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:06:04.907685       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:12:14.910310       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:18:59.912786       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:28:26.915097       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:36:07.918068       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:42:21.920765       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:50:32.922858       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:59:41.925302       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:08:59.928203       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:17:28.931148       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:26:46.933553       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:34:09.935912       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:43:55.938552       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:53:39.941393       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:58:48.943846       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:05:18.946504       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:11:02.949306       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:19:57.952531       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:25:25.954797       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:34:08.957314       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:39:20.959650       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:46:39.962231       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:51:48.964441       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:58:44.967059       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:06:50.969978       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:14:03.972582       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:19:59.975390       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:27:37.977866       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:35:05.980603       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:42:32.984005       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:51:33.986014       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:56:42.988986       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:03:56.992089       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:13:37.994288       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:21:38.996435       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:27:22.998952       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:32:48.001723       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:42:45.004101       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:47:50.007129       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:56:59.010053       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:05:12.012986       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:12:25.015687       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:20:06.018931       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:27:32.021050       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:35:27.024038       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:40:49.026714       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:47:56.028927       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:54:38.031681       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:03:41.034147       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:10:12.036762       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:18:08.039613       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:27:22.042375       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:35:36.044927       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:43:23.047672       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:52:18.050472       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:01:22.052697       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:06:24.055484       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:15:04.057972       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:22:23.060640       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:30:57.063684       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:39:12.066416       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:45:26.068692       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:52:01.071362       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:57:09.074227       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:05:44.076825       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:12:16.079154       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:17:54.081290       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:27:44.084449       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:37:31.086924       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:43:14.089806       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:51:29.092103       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:00:06.094320       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:06:53.097234       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:12:08.099891       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:21:34.102903       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:27:39.105586       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:35:49.108139       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:42:25.111048       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:50:03.113115       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:56:35.115450       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:06:12.118748       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:12:51.121362       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:20:46.124167       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:27:11.127448       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:34:31.130522       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:40:42.132752       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:47:14.136708       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:52:41.138642       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:02:21.141474       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:10:42.143766       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:18:46.146531       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:28:00.149452       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:35:46.151890       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:44:13.154193       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:53:16.156327       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:00:55.158640       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:09:15.161748       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:16:13.164902       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:22:11.166896       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:29:06.169440       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:34:26.171947       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:41:59.174585       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:47:58.176824       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:53:08.178741       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:58:44.181449       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:07:38.184565       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:12:44.187578       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:18:17.189876       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:27:34.193042       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:36:56.195104       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:46:50.197378       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:52:19.200305       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:57:24.202857       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:03:41.206358       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:10:42.209374       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:17:08.211787       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:25:00.214684       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:30:49.217988       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:39:55.220193       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:45:02.222307       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:52:47.224833       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:59:13.226714       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:05:56.228962       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:14:47.231500       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:19:54.233977       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:27:16.236307       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:32:33.238905       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:38:24.241569       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:44:16.243870       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:50:08.246236       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:57:00.248673       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:02:05.250899       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:10:40.253126       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:15:48.255687       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:24:04.258051       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:33:34.260118       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:42:18.262902       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:51:55.264858       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:58:35.266810       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:04:01.269679       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:09:37.272075       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:16:45.274343       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:21:45.276933       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:27:54.279036       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:35:43.281344       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:40:45.284094       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:49:49.287072       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:56:31.289301       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:06:05.292039       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:11:51.294930       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:21:46.298003       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:30:40.300080       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:39:47.302438       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:45:53.305285       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:51:53.307880       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:59:12.310641       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:05:40.312938       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:11:11.315911       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:16:26.319252       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:25:54.322068       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:35:38.324468       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:43:41.326256       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:52:53.329122       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:58:31.331219       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:07:45.333750       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:15:44.336652       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:24:33.338632       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:30:31.341002       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:39:23.343673       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:47:39.346354       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:57:27.349217       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:04:45.351912       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:13:06.353979       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:22:37.356378       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:27:58.359328       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:35:05.361665       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:43:26.364027       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:50:45.365903       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:00:38.368094       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:05:40.370235       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:11:52.373259       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:16:57.375559       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:21:59.377724       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:31:52.379970       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:36:53.382210       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:43:19.384598       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:49:04.386717       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:57:06.389011       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:03:20.392269       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:09:29.394752       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:17:50.397336       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:26:25.400160       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:34:47.402783       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:42:20.404927       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:49:52.407839       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:59:01.410903       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:04:13.413677       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:11:38.415856       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:19:26.418655       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:25:29.420732       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:30:30.423519       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:38:01.426368       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:45:48.429311       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:54:36.432493       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:02:56.435437       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:08:59.438583       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:18:17.443057       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:24:47.445907       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:31:24.447876       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:40:12.450277       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:49:59.452814       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:59:20.455703       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:07:49.458079       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:16:22.460290       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:21:23.463447       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:31:12.465889       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:36:13.468059       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:45:36.472561       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:51:25.475319       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:00:18.477252       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:07:33.480176       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:17:07.484125       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:22:13.488241       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:27:53.490592       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:37:52.493599       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:47:09.495675       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:56:12.498044       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:05:07.499979       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:11:29.502563       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:20:10.504895       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:27:35.506846       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:36:04.509904       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:44:38.513276       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:53:34.515355       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:02:27.517894       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:09:14.519950       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:18:37.522432       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:24:30.525453       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:32:33.527778       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:40:24.530655       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:46:47.532911       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:52:45.535929       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:01:43.538440       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:06:43.542320       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:13:56.544864       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:21:08.547220       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:29:25.549514       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:34:34.554756       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:43:01.557272       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:48:59.559508       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:58:09.564801       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:04:18.566710       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:11:34.569617       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:17:40.572350       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:27:03.574760       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:36:28.577047       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:42:16.579755       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:49:16.581997       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:56:33.584261       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:02:16.586712       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:07:57.589306       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:16:20.592302       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:25:05.594699       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:32:04.597271       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:38:45.600176       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:48:43.603209       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:57:21.605557       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:06:07.608497       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:13:53.612473       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:19:47.614728       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:29:29.617483       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:39:20.620886       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:46:32.624624       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:52:05.627609       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:01:34.629671       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:07:03.632731       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:13:52.635497       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:21:22.638210       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:30:51.640491       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:37:52.643382       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:46:35.645612       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:56:33.648542       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:05:08.651648       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:14:51.654589       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:20:06.657005       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:30:02.660415       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:36:03.663046       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:43:22.665955       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:48:25.668936       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:54:23.671177       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:01:17.674071       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:06:22.677108       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:13:27.679660       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:20:03.682077       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:26:38.685045       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:31:56.687634       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:41:35.689882       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:47:54.692811       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:56:50.696039       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:04:48.698561       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:10:34.701513       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:16:31.704384       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:23:59.706290       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:29:16.709207       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:34:59.711907       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:41:21.715108       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:48:18.718138       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:55:04.721255       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:04:21.723811       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:12:04.726720       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:17:15.728791       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:24:17.731184       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:29:49.732987       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:37:25.735499       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:46:42.738062       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:53:16.741386       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:03:08.744801       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:10:10.747709       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:16:06.750616       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:21:51.752776       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:29:54.754761       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:38:57.756956       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:46:44.759606       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:51:55.761744       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:58:11.764297       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:04:43.767103       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:10:48.769425       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:17:02.771496       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:25:14.774120       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:32:18.775981       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:40:52.778677       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:46:11.780788       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:54:09.783778       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:01:13.786430       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:06:42.788811       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:15:51.791440       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:23:11.793862       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:30:55.796177       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:38:09.798547       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:47:51.801834       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:56:04.804289       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:03:49.806780       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:10:59.808869       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:20:43.811568       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:26:32.813496       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:33:36.816097       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:42:35.818295       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:48:01.820983       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:56:07.822945       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:01:25.825168       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:06:25.827964       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:13:14.830241       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:22:33.832860       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:30:09.835782       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:37:46.837858       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:46:03.839930       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:53:49.842490       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:01:44.845375       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:10:17.848148       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:19:09.850596       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:24:36.853156       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:31:47.856202       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:38:08.858929       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:46:38.861377       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:52:45.864542       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:02:11.866969       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:11:40.869650       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:20:51.871596       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:26:50.874463       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:32:40.876341       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:38:07.878592       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:44:49.881421       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:51:52.884031       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:57:00.886180       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:04:02.888080       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:12:39.890431       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:19:39.892576       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:28:49.895168       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:36:43.897133       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:45:49.899986       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:51:05.901911       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:57:22.904504       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:06:04.906690       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:13:51.909314       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:22:23.911751       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:31:39.914013       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:37:23.916956       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:47:04.919382       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:52:35.921943       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:00:50.923985       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:06:11.926332       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:11:40.929000       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:20:29.930940       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:27:59.933733       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:37:36.936811       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:42:46.939499       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:49:12.942167       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:55:11.944292       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:01:24.946344       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:07:25.949130       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:15:51.951840       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:23:24.954361       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:30:02.956750       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:35:27.959337       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:44:38.962876       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:52:34.964921       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:01:19.967913       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:06:38.970680       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:12:34.972824       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:22:12.976161       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:28:40.978975       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:36:01.981159       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:43:49.983512       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:50:25.986211       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:56:50.988153       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:03:40.990263       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:10:42.993240       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:17:07.996040       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:24:31.998703       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:34:13.001922       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:39:29.004278       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:46:19.006053       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:54:52.009118       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:01:39.011431       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:09:40.014061       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:17:21.016968       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:26:04.019203       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:32:17.021242       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:41:54.023933       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:49:17.026836       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:58:50.029629       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:04:05.032123       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:11:04.035349       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:16:28.037871       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:22:31.040462       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:27:52.042984       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:33:05.045440       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:39:49.047550       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:49:37.050124       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:59:33.053393       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:06:08.055965       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:13:13.059284       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:18:13.061729       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:24:57.064054       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:34:20.067104       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:40:39.070177       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:46:19.072262       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:55:01.074387       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:00:40.076837       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:07:26.078938       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:13:42.081768       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:21:27.084523       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:27:14.087073       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:33:10.089458       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:42:48.092326       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:50:19.094883       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:55:54.097918       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:01:37.100688       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:09:28.103792       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:17:49.106808       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:26:41.110048       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:34:45.112968       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:43:35.115360       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:50:05.118141       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:56:37.120446       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:03:41.123475       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:11:52.126296       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:17:03.128816       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:23:57.131511       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:28:57.133989       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:38:09.136707       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:46:53.139282       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:54:20.141491       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:02:46.143657       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:11:25.146921       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:19:57.149499       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:26:56.151965       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:32:10.154834       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:39:29.157484       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:45:48.160516       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:55:24.163194       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:01:03.165588       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:08:44.168323       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:14:24.171354       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:21:54.174355       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:28:23.176674       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:34:14.179560       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:43:49.181796       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:53:23.184274       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:02:10.187665       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:07:42.190480       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:16:45.195998       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:05.199207       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:36:03.202946       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:42:01.205768       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:50:37.208785       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:21.211023       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:03:16.213597       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:11:51.215779       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:21:50.218290       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:28:45.220725       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:34:02.223007       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:42:30.224794       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:51:52.226993       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:57:37.230438       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:04:23.233026       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:04.235707       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:19:21.238748       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:27:48.241310       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:34:40.244344       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:39:45.247257       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:45:37.250662       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:51:02.253983       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:00:11.256710       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:08:41.259001       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:17:54.261934       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:27:17.264651       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:34:58.266875       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:44:40.270141       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:53:15.273537       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:59:17.276498       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:06:53.279303       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:13:19.282202       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:20:20.284281       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:29:57.287019       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:36:07.289572       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:41:15.291900       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:49:01.294313       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:55:02.296884       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:04:53.300049       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:13:08.303294       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:21:13.306093       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:28:54.308208       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:35:09.310213       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:40:55.312291       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:49:09.314770       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:55:43.317320       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:04:17.319985       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:09:29.323079       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:16:04.325726       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:25:38.328567       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:33:10.331494       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:41:16.333891       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:49:46.336124       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:59:06.339085       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:06:04.341528       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:15:56.344442       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:24:34.347007       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:33:23.349530       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:41:55.352476       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:48:07.354584       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:56:28.357116       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:04:45.359722       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:11:23.362235       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:17:47.365676       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:26:14.368961       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:34:11.372304       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:40:09.375019       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:50:06.377784       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:56:57.380492       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:02:08.383769       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:11:15.386484       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:17:38.388850       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:24:28.391757       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:33:24.394806       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:41:42.398002       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:49:55.400626       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:56:00.402826       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:02:39.404808       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:12:22.407154       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:19:16.410287       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:25:35.412664       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:32:04.414878       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:37:55.417250       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:45:57.419863       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:52:21.422965       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:00:35.425271       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:06:04.427889       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:12:39.430120       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:20:43.432981       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:25:59.435135       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:35:15.438118       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:42:51.440153       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:52:26.444319       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:59:02.445918       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:08:31.448939       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:18:19.451609       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:24:13.454292       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:32:36.457279       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:38:24.459380       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:44:10.461982       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:51:00.464249       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:56:03.466578       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:04:45.469273       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:12:41.473054       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:21:04.475884       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:29:25.477813       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:37:06.480925       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:42:15.483860       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:49:48.486311       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:56:06.488813       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:01:19.491315       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:07:51.493927       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:14:43.495865       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:23:36.498352       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:31:22.500942       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:37:59.503609       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:46:43.506153       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:54:55.508272       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:03:39.511226       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:12:55.514214       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:21:39.517187       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:30:47.519795       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:40:06.522063       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:47:58.525143       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:53:31.527596       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:59:45.529787       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:09:29.532707       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:15:29.535259       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:22:09.538223       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:31:20.541025       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:37:01.543060       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:46:52.546154       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:55:59.549087       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:04:21.551722       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:11:22.554303       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:17:31.557626       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:25:35.560649       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:35:04.562977       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:44:35.565228       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:50:56.567642       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:59:20.570213       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:08:04.572831       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:15:32.575736       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:22:48.578893       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:28:55.580773       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:36:59.583567       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:42:48.585612       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:52:27.587549       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:59:31.590502       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:08:17.593264       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:16:57.595762       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:22:34.597955       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:29:37.599884       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:37:42.602196       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:43:12.604549       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:52:45.607646       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:58:40.610296       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:06:42.614052       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:12:59.616128       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:18:04.618601       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:27:22.620801       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:36:35.622896       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:43:49.625181       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:53:08.628044       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:01:58.630690       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:07:16.633374       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:16:35.636307       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:24:45.639148       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:32:43.641106       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:40:12.643600       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:47:42.646815       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:56:23.649671       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:05:43.652014       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:15:07.654345       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:22:39.656354       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:31:22.658900       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:40:00.661149       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:45:56.663846       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:54:46.667119       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:01:59.669488       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:07:50.671906       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:16:14.674756       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:24:00.677302       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:32:00.679898       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:39:52.682434       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:48:27.685286       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:58:01.688480       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:06:20.690686       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:11:51.692540       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:20:19.694689       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:27:44.698361       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:36:36.701681       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:42:52.703951       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:51:27.706499       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:57:20.708811       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:04:01.712269       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:09:08.714685       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:16:29.717622       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:22:33.720021       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:27:59.722201       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:36:50.725146       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:43:01.727341       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:49:27.729987       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:58:46.733094       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:06:23.735547       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:11:30.737552       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:17:47.740001       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:26:43.743275       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:32:07.745485       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:40:56.748491       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:50:39.751040       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:58:38.753230       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:05:39.755926       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:11:33.758358       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:17:46.761278       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:26:17.763989       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:36:01.766629       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:45:49.769499       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:55:30.771392       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:00:48.772945       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:06:27.775785       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:13:32.778496       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:21:04.781159       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:30:52.783445       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:39:06.786587       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:44:53.789693       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:51:15.792288       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:58:45.794778       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:04:17.797902       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:11:13.800246       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:18:00.802193       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:26:59.804940       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:36:39.808079       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:43:15.810694       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:51:25.813433       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:59:06.816319       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:04:28.818845       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:13:00.822067       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:22:22.824798       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:29:59.827181       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:35:49.830127       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:43:50.833340       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:51:08.836054       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:59:26.838687       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:08:07.841697       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:17:40.844355       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:27:36.847712       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:33:51.850435       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:43:43.853848       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:51:49.856843       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:00:27.859946       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:08:29.862820       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:16:18.865465       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:25:15.867989       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:32:50.870443       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:39:00.872662       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:46:50.874851       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:56:32.877822       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:02:58.879921       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:08:10.883181       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:14:05.885830       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:22:50.888817       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:31:36.891208       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:37:18.893562       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:47:02.895889       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:54:33.898933       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:01:08.902294       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:07:08.904758       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:16:15.906627       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:22:28.909552       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:29:04.912898       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:38:31.916139       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:47:46.918156       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:53:24.920135       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:02:03.922644       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:09:56.924758       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:19:46.927620       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:26:52.929757       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:35:29.932318       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:40:49.934919       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:45:52.937334       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:52:19.939748       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:58:10.942030       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:06:10.945021       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:13:41.947475       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:18:41.951582       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:27:57.954664       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:33:39.956825       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:42:40.958527       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:48:46.961363       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:54:04.963260       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:01:45.966658       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:07:40.969130       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:15:11.972011       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:22:44.974651       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:30:01.977794       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:39:07.980836       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:47:00.983454       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:52:27.986209       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:59:18.989297       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:08:44.991465       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:18:26.993923       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:26:36.996597       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:36:17.999068       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:44:18.001453       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:49:20.003975       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:58:40.006364       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:07:14.009155       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:12:18.011940       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:20:51.014368       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:30:20.017481       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:37:57.020646       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:46:47.023490       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:53:08.025981       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:58:52.028571       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:07:07.030836       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:12:24.033074       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:18:01.036110       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:26:17.038713       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:33:03.042113       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:42:30.044214       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:50:39.046172       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:57:59.049258       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:07:40.052188       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:16:17.054271       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:22:30.057389       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:29:32.060009       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:38:39.062357       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:44:45.065183       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:53:10.068213       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:58:53.070996       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:04:16.073765       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:13:03.076207       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:20:08.079603       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:26:39.082047       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:32:46.085260       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:38:30.088513       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:44:21.090735       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:51:28.093610       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:56:50.096042       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:06:40.098937       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:11:59.100931       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:21:46.103948       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:27:46.107062       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:33:07.109849       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:42:25.113026       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:47:50.116205       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:54:13.118639       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:03:52.121918       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:10:34.125175       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:18:56.127741       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:27:36.130334       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:34:53.133322       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:44:41.136018       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:51:04.138088       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:00:31.140306       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:08:04.143073       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:13:36.145918       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:22:06.148654       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:29:18.151361       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:34:47.154196       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:43:49.156677       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:50:27.160131       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:55:40.162662       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:01:50.165244       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:10:33.168533       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:19:27.171736       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:26:06.174553       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:31:34.177633       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:41:02.180762       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:46:02.182990       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:55:04.185364       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:00:27.188128       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:06:28.190055       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:12:47.192359       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:18:10.195064       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:25:49.197898       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:33:54.200885       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:40:47.203763       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:46:25.206504       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:51:36.208120       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:59:59.210792       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:06:57.213966       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:13:51.216109       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:21:40.218958       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:27:53.221828       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:36:34.224206       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:43:06.227118       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:51:20.229781       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:59:48.232648       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:08:42.234818       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:16:02.237374       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:22:05.240220       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:31:28.242366       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:38:40.246259       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:46:57.249597       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:56:13.252584       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:03:41.255528       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:12:23.258494       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:18:30.261164       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:23:47.263624       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:29:46.269722       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:36:51.272575       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:45:43.274997       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:54:07.277621       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:03:31.280442       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:10:47.283507       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:15:50.286325       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:21:07.288890       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:30:16.291745       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:37:55.294763       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:43:38.296908       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:49:33.299696       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:58:42.302868       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:07:05.305289       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:15:00.308470       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:24:11.311004       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:31:08.313181       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:39:17.315343       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:45:24.318343       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:50:37.320802       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:00:01.323452       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:08:37.326341       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:15:10.328494       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:21:06.331508       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:30:06.334083       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:35:49.336302       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:41:32.339097       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:50:14.341757       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:59:53.345040       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:07:19.347562       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:12:57.350049       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:21:14.352907       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:30:16.355450       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:36:03.358673       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:45:05.361184       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:53:23.363604       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:02:36.367026       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:08:45.369177       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:17:37.371467       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:27:25.373724       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:37:19.375990       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:42:27.379032       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:50:17.381267       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:57:38.384169       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:07:19.386954       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:13:05.389645       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:20:09.392268       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:25:25.395346       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:33:22.397982       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:41:28.400144       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:48:55.402515       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:54:13.404468       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:01:02.407298       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:07:51.410036       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:13:15.412379       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:21:30.414901       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:30:06.417158       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:37:20.419713       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:44:23.422551       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:49:48.425535       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:55:09.428376       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:04:48.430643       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:11:47.432905       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:17:16.435222       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:24:57.437492       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:30:23.440021       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:39:15.443607       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:47:48.445638       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:56:59.448057       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:04:16.450478       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:12:47.452307       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:20:54.454774       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:30:41.456762       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:39:16.458986       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:45:14.461707       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:54:30.463874       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:00:07.466261       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:08:51.469392       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:16:27.472009       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:24:14.474785       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:32:16.476845       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:40:23.478865       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:50:16.481954       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:55:48.484810       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:03:17.486648       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:08:56.489800       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:17:48.491723       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:27:23.494158       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:37:16.496802       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:44:57.499899       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:53:30.502540       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:00:37.504738       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:06:58.506918       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:14:02.509138       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:23:34.511130       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:30:42.513218       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:39:42.515885       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:45:42.517927       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:53:41.520816       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:02:21.523600       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:10:16.527066       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:18:44.529123       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:25:29.531613       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:33:36.534174       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:42:08.536247       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:51:53.538530       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:59:29.541205       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:07:20.543727       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:13:09.546254       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:22:34.548950       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:29:28.551759       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:38:20.554656       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:46:56.557806       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:51:56.560231       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:58:52.562977       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:08:30.565110       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:18:09.568349       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:27:28.571390       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:35:08.573688       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:43:19.575753       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:50:14.578029       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:59:31.580675       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:05:40.582817       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:11:50.585389       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:20:40.588069       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:26:48.589713       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:34:32.592565       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:39:37.595501       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:49:24.598885       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:58:32.601740       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:04:03.604123       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:11:02.606512       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:17:59.608783       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:24:42.611819       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:34:36.614295       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:44:31.616102       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:49:45.619029       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:57:59.621548       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:05:35.624522       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:11:21.627001       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:18:04.629738       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:24:47.632167       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:34:26.634561       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:43:37.637323       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:51:15.639874       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:56:42.642499       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:04:10.645496       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:13:37.648619       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:19:46.650497       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:28:48.653474       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:36:37.656496       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:45:04.658666       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:52:26.661740       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:01:47.664212       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:10:36.666904       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:18:34.670314       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:26:31.673555       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:33:57.675509       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:38:57.678466       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:47:02.681626       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:55:49.684044       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:04:01.686107       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:09:27.688748       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:16:39.691296       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:24:43.693578       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:33:42.696800       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:43:32.699169       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:49:15.702028       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:59:08.705048       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:07:52.707560       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:15:57.710000       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:25:21.711685       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:31:56.714049       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:39:47.717485       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:46:30.720514       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:53:17.722859       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:02:51.725219       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:12:27.728319       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:21:51.731099       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:30:53.733716       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:38:38.736689       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:45:34.739078       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:52:45.741641       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:01:56.744571       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:07:28.747173       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:17:24.749759       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:26:52.752062       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:32:51.754790       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:41:45.757334       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:50:17.761105       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:57:01.763304       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:04:54.766153       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:13:59.769488       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:20:16.772464       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:27:25.775379       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:35:12.778069       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:43:26.780761       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:49:00.783197       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:58:19.785814       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:06:43.788262       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:16:40.790873       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:22:40.793196       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:29:05.795686       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:36:33.798358       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:42:16.801242       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:49:09.803960       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:56:55.807395       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:06:25.810445       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:11:58.813267       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:21:12.815577       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:29:01.818786       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:38:57.821358       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:44:32.824091       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:50:52.826847       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:00:10.829595       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:06:43.832939       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:13:14.835853       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:20:25.838433       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:27:22.840735       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:35:39.843630       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:45:34.846123       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:54:28.848691       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:01:18.851508       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:09:20.854003       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:17:12.857112       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:27:00.859589       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:36:22.862778       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:46:08.865653       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:52:55.868112       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:00:41.870653       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:08:48.882574       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:17:30.885468       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:23:21.888545       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:30:42.891752       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:37:05.894175       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:45:57.896779       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:54:34.899202       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:02:23.902030       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:11:22.905031       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:21:03.908225       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:30:39.910988       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:36:29.914065       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:44:52.916475       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:51:19.919052       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:59:32.921531       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:09:07.924095       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:18:28.926488       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:26:04.929723       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:33:35.932974       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:42:35.935761       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:51:27.939052       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:57:12.942070       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:04:05.944869       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:12:28.947134       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:20:44.950038       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:29:38.953250       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:35:24.955735       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:43:13.958660       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:51:58.962231       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:59:28.964445       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:06:21.966927       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:13:34.969312       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:20:01.972559       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:27:20.975046       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:33:15.977223       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:40:19.980526       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:46:59.983284       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:52:26.985777       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:59:40.988371       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:06:43.992520       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:14:34.994864       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:20:58.997904       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:28:07.001086       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:37:47.004341       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:45:29.007757       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:52:28.010585       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:59:26.012951       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:05:50.015332       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:13:33.018300       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:22:39.020798       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:28:48.023914       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:38:14.026479       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:45:44.028897       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:54:20.031571       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:01:55.034062       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:08:18.036663       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:16:50.039785       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:23:50.041699       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:32:11.044023       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:41:21.046865       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:51:08.049612       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:57:18.052877       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:06:14.055590       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:15:17.057908       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:24:50.060262       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:30:46.062560       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:37:42.065552       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:47:14.067975       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:56:57.071196       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:05:24.073767       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:12:31.076257       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:20:22.078775       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:29:04.082002       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:35:48.085332       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:43:30.088101       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:53:16.091269       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:01:37.094612       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:10:35.097511       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:16:42.100741       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:23:33.102907       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:29:54.105394       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:36:19.107898       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:45:06.110741       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:52:02.113934       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:00:58.117121       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:10:23.119761       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:17:09.123163       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:23:00.126309       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:30:18.128430       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:38:50.130810       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:47:04.133335       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:53:47.136162       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:02:05.138515       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:10:18.142127       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:15:21.144384       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:21:06.146164       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:29:59.148533       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:37:03.151150       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:46:59.153999       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:56:40.156825       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:06:33.159564       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:14:17.161376       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:23:00.164162       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:28:33.166457       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:35:58.169210       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:41:54.171799       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:50:27.174039       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:55:47.175768       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:01:41.178697       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:07:50.181061       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:15:47.183445       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:25:46.185978       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:35:04.188572       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:41:39.191565       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:48:40.193857       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:56:36.196021       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:04:31.199123       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:12:25.202358       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:20:55.205367       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:27:07.208051       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:33:07.211245       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:40:52.214129       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:49:43.217354       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:55:06.220894       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:02:47.224282       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:10:45.226975       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:20:28.229737       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:25:46.232322       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:34:33.235065       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:40:43.237964       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:48:02.240378       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:54:16.243029       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:59:21.246298       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:08:48.248602       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:18:23.250681       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:25:14.252953       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:31:47.255805       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:41:19.258456       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:51:13.261754       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:59:43.264735       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:08:16.267279       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:16:09.269646       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:23:37.272507       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:31:09.275306       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:38:36.278008       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:47:32.280351       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:53:05.283548       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:00:09.286196       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:09:01.291764       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:14:23.296171       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:22:31.299337       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:28:43.301982       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:36:24.304608       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:42:59.306891       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:52:46.309657       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:00:32.312030       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:10:24.314222       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:17:23.317191       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:23:17.319992       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:31:46.322976       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:39:50.325898       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:48:58.328891       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:55:56.331265       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:01:28.333941       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:07:23.336550       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:13:39.339529       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:19:58.342586       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:29:28.345800       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:36:49.348387       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:45:14.350885       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:50:51.353249       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:57:49.355303       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:05:38.357684       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:12:04.360911       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:19:24.364081       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:27:11.366316       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:34:37.368502       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:40:43.371448       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:47:04.373962       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:53:38.377596       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:00:28.380288       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:05:40.382711       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:10:49.385697       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:17:56.388793       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:26:52.391804       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:36:51.394803       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:42:18.396972       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:47:26.399469       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:55:30.402039       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:02:09.405594       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:10:15.408939       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:15:21.411090       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:25:05.413045       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:30:19.416039       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:37:37.418693       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:46:41.421362       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:53:38.423277       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:00:24.425701       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:05:33.428012       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:13:05.430112       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:18:30.433385       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:25:58.435734       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:35:15.437917       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:44:49.442896       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:52:09.445296       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:01:51.447540       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:11:07.449677       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:18:33.451744       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:26:48.454128       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:36:16.457122       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:45:55.459635       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:54:24.462661       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:01:46.464915       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:08:07.467777       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:14:31.470761       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:21:19.473298       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:29:51.475953       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:38:57.478693       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:45:13.480860       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:51:11.483207       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:00:23.486130       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:07:39.489093       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:14:55.491469       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:21:29.493643       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:28:11.496370       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:34:02.498869       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:41:55.501245       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:49:18.503727       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:58:01.505990       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:05:09.508595       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:13:46.511295       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:22:29.513580       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:30:57.515863       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:40:14.518235       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:48:16.520802       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:53:50.523168       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:01:46.526112       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:08:18.528835       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:14:01.531176       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:21:05.534759       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:28:10.538039       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:33:30.540849       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:41:01.543693       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:49:37.546153       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:58:08.549210       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:04:51.551710       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:14:43.554339       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:24:17.557577       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:29:55.560656       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:39:41.563381       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:46:27.567289       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:52:45.570114       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:58:52.572530       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:06:58.574867       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:16:12.577212       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:26:09.579517       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:34:54.582074       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:44:01.584676       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:50:54.587011       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:57:32.590238       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:02:48.592689       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:08:50.595888       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:14:39.598338       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:24:08.600956       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:31:02.603735       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:39:22.606513       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:44:41.608996       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:53:10.612237       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:01:56.615511       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:07:36.617283       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:15:45.620519       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:22:54.622952       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:31:55.625682       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:41:53.628848       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:47:21.631271       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:54:18.633499       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:03:55.636340       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:11:32.639473       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:21:17.643033       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:27:08.645129       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:34:18.648113       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:40:47.651072       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:48:21.654464       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:54:59.656911       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:02:11.660150       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:07:39.662454       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:13:44.664582       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:22:05.667010       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:27:28.669596       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:32:43.672062       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:39:43.674676       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:46:34.677498       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:54:43.680677       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:02:53.683660       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:11:12.686299       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:20:37.689052       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:28:18.692096       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:33:40.694543       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:43:28.697022       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:52:22.699086       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:01:21.700974       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:10:53.703574       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:19:52.705687       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:26:05.708916       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:34:21.712152       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:41:45.714533       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:51:39.716716       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:00:12.719078       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:09:11.722277       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:17:57.724320       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:23:28.726884       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:32:29.729347       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:38:43.731817       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:46:36.733655       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:53:25.736707       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:59:27.739416       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:07:20.741749       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:13:42.743974       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:23:40.745887       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:32:51.748053       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:40:34.750773       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:48:12.753528       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:56:21.756252       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:03:05.759193       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:11:04.762277       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:16:16.764887       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:23:55.767129       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:33:19.770369       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:39:50.772868       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:48:51.776469       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:54:35.778764       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:00:50.781374       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:07:24.784159       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:16:14.786771       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:23:06.789432       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:29:32.792358       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:39:02.795071       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:44:08.797268       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:51:36.799799       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:59:04.802167       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:07:13.805382       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:17:12.807971       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:25:59.810517       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:35:00.813518       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:43:56.816552       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:53:50.819526       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:01:06.822648       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:06:46.825657       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:15:09.827846       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:23:20.830941       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:31:25.833768       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:41:20.836046       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:47:33.838112       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:54:06.840196       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:02:42.843563       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:09:33.846032       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:18:48.849096       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:27:37.851480       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:33:23.854215       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:42:33.857142       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:52:05.860257       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:59:44.862695       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:09:40.866110       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:17:12.868223       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:23:48.871280       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:31:20.874117       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:40:56.876209       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:50:49.879016       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:56:12.881031       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:01:21.882790       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:06:38.885265       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:12:35.888414       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:20:28.890608       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:29:13.893177       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:36:55.896242       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:45:39.899238       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:51:54.902104       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:57:15.904490       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:03:56.906971       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:10:47.909774       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:18:49.912740       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:25:47.914649       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:33:29.917148       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:39:19.919493       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:46:05.922292       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:54:42.925186       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:04:13.928032       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:13:51.931258       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:21:53.934240       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:29:13.937231       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:34:56.939879       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:41:58.942517       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:49:24.945047       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:59:01.947622       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:08:44.951001       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:15:27.953584       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:23:40.956044       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:32:11.958664       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:38:51.961111       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:48:15.963451       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:55:38.966110       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:00:56.968943       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:08:52.971550       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:14:02.974244       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:23:07.977090       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:32:57.979982       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:41:22.982854       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:48:44.986024       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:54:05.988983       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:59:16.991503       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:06:43.994298       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:15:25.996771       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:23:57.999296       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:32:10.002351       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:38:16.004863       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:47:17.007496       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:55:33.010007       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:01:19.013164       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:08:28.015896       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:16:12.018140       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:22:17.020657       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:28:14.023912       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:36:19.026516       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:44:40.029754       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:50:58.032153       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:00:14.034491       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:08:32.037438       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:18:10.040059       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:25:20.042497       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:30:30.045061       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:36:42.047166       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:43:05.049203       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:48:22.051626       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:56:08.054716       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:05:54.057746       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:12:59.060835       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:18:18.063655       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:26:58.066107       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:35:28.069434       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:41:05.072027       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:48:10.074772       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:56:14.078193       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:03:11.080928       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:08:25.083773       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:14:33.086428       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:20:24.089025       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:27:19.090990       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:36:56.093457       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:44:03.095814       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:51:25.098567       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:57:59.101774       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:03:05.104232       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:09:33.106648       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:16:20.109930       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:22:40.113255       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:29:47.115885       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:38:15.118851       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:44:29.121964       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:53:41.124872       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:02:17.127268       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:10:13.130150       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:16:29.133834       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:26:21.136224       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:34:58.139707       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:40:49.142817       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:48:33.145377       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:54:34.149827       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:01:28.153042       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:08:06.154931       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:16:38.157342       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:22:27.159730       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:30:55.162825       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:36:03.165704       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:43:51.168388       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:49:07.171643       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:58:42.174813       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:03:57.176666       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:09:25.179537       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:14:26.181899       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:23:06.184469       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:31:24.187123       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:37:09.189762       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:43:05.191616       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:52:51.194726       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:58:32.196956       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:04:18.199440       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:12:39.202190       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:22:10.205142       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:27:42.207743       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:33:15.209835       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:39:30.211643       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:47:20.214177       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:56:51.217095       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:04:37.219755       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:11:20.222940       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:18:57.225279       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:28:22.227979       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:37:55.229766       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:43:13.232623       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:52:47.234632       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:02:21.237299       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:10:22.239800       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:18:05.242911       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:27:11.245767       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:32:11.248201       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:39:29.250757       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:48:04.253448       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:56:43.255478       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:02:27.257867       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:11:02.260481       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:17:33.264288       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:26:08.267201       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:35:53.270254       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:44:51.272931       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:54:12.275721       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:00:12.279005       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:05:30.281715       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:15:16.284710       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:23:55.287497       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:33:13.290272       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:39:42.293689       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:46:23.296978       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:54:48.300177       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:04:42.303460       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:12:29.306395       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:21:08.309330       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:29:53.312221       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:39:01.314837       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:44:48.318002       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:50:28.320845       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:56:02.323287       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:05:45.326511       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:12:20.329278       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:20:05.340539       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:30:02.343128       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:36:39.345272       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:45:32.347645       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:55:20.350181       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:04:44.352954       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:13:05.355592       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:19:53.358258       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:25:32.360920       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:33:26.363167       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:39:19.365459       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:46:02.368356       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:52:40.371088       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:01:28.373576       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:10:13.375913       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:17:36.378703       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:22:41.381734       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:31:29.384274       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:39:03.386860       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:48:03.390135       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:56:50.392330       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:05:11.394715       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:10:51.396751       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:19:12.399881       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:28:39.402865       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:34:45.406018       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:41:26.408215       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:51:18.410637       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:00:16.413129       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:08:11.416209       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:14:30.418213       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:20:01.421634       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:25:07.424587       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:32:51.427858       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:40:04.430354       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:49:27.432243       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:58:37.434799       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:03:56.437032       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:13:32.440321       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:18:41.445376       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:24:55.447296       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:34:40.449332       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:43:52.451765       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:52:35.453900       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:59:45.456237       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:05:10.458444       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:10:38.460951       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:18:47.463988       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:27:55.466350       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:35:31.469306       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:40:33.471785       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:50:02.474835       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:57:43.477894       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:07:08.480126       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:14:23.482348       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:20:15.484268       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:26:27.486892       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:35:02.489569       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:44:20.491768       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:50:59.494334       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:59:28.496551       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:09:10.499574       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:18:19.502616       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:23:39.505712       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:28:49.511718       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:35:20.514502       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:44:27.517613       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:51:35.519783       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:56:47.522395       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:02:39.525529       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:09:02.527664       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:18:38.530270       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:27:09.532465       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:34:49.536209       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:43:57.539547       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:51:15.542530       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:00:14.544354       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:09:20.546342       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:16:18.549349       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:21:33.551386       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:30:42.554512       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:37:48.556694       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:47:42.559584       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:55:47.562518       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:03:45.565076       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:09:39.567123       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:17:38.569717       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:25:12.572289       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:34:51.574508       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:41:43.577084       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:49:58.579559       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:57:42.581495       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:05:48.583848       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:11:37.586941       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:20:32.589380       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:30:14.592138       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:39:49.595230       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:47:36.597799       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:55:07.600724       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:05:03.603782       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:14:07.606380       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:24:03.609161       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:32:00.611690       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:38:00.614324       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:43:46.616528       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:49:58.618638       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:55:50.621465       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:03:22.627232       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:13:11.629745       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:22:47.632117       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:32:43.634717       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:41:26.637137       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:47:08.640027       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:54:10.642738       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:02:08.644661       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:11:30.647388       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:19:19.649865       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:27:58.652302       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:33:44.655498       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:39:30.658061       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:47:23.660160       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:57:01.663203       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:05:37.666465       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:13:29.669386       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:18:35.672772       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:24:52.676000       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:32:49.678486       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:41:07.681309       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:47:19.683546       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:56:05.686109       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:04:30.688245       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:09:53.690639       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:14:56.692421       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:20:27.694449       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:25:27.697305       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:31:45.700154       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:38:03.702519       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:44:03.705178       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:53:37.708522       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:58:58.711188       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:04:33.713983       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:10:36.716706       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:16:05.719672       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:22:38.722151       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:28:12.724122       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:36:05.726576       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:43:05.729128       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:50:20.732038       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:58:48.734596       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:08:30.736957       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:17:15.739283       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:22:40.741483       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:29:02.743246       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:38:02.746290       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:44:41.748513       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:52:09.751102       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:58:00.753865       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:05:20.755859       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:11:46.758002       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:21:36.760693       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:30:53.763252       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:38:24.765486       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:44:35.768286       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:52:52.771225       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:00:35.774223       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:06:57.776435       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:13:13.779208       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:18:23.781712       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:27:12.784536       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:34:14.787024       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:42:44.790011       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:49:50.793036       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:55:05.795096       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:04:26.797632       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:10:22.800163       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:16:59.802832       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:26:55.804863       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:36:30.807563       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:43:58.810000       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:51:28.812081       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:57:07.814901       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:06:43.817862       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:13:11.820120       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:20:15.822146       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:25:28.824936       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:34:12.827638       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:40:04.830011       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:48:39.832040       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:56:05.834642       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:01:53.836607       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:07:56.838267       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:17:26.840791       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:23:11.843857       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:32:29.845958       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:41:42.848600       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:48:03.850478       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:53:59.852974       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:03:42.855465       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:12:35.857571       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:21:35.860506       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:29:27.862855       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:36:55.865479       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:43:30.868523       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:51:57.871522       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:58:30.874001       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:06:44.876389       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:14:28.879108       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:19:40.881816       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:27:14.884676       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:33:49.887168       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:41:35.889769       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:48:27.892466       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:54:32.896329       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:02:41.898661       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:11:16.901207       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:18:49.903056       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:26:09.904714       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:35:49.906878       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:43:27.909702       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:49:01.912911       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:56:15.915531       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:05:38.918224       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:15:22.920812       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:24:41.923928       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:33:23.926439       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:42:51.929147       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:52:18.932193       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:01:59.934946       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:08:16.937541       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:14:32.940199       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:21:13.942697       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:31:12.945530       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:40:10.948044       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:46:29.950614       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:56:19.953388       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:06:14.956255       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:12:58.958395       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:22:52.960998       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:30:49.962937       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:39:44.965658       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:47:30.967776       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:52:34.969566       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:59:18.972451       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:06:59.975580       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:14:53.978128       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:23:48.980925       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:30:32.983651       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:36:37.985829       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:44:02.988661       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:52:39.990879       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:57:49.993429       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:07:01.996047       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:16:42.998627       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:21:45.000576       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:28:25.003479       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:34:26.006723       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:43:05.009345       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:49:27.011943       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:56:28.014974       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:02:26.017540       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:10:31.020156       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:19:16.022801       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:28:26.025743       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:37:05.028164       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:44:39.029850       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:52:52.032309       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:02:45.035252       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:08:23.038345       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:13:24.041492       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:23:23.044495       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:31:43.046987       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:36:45.050029       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:46:44.052634       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:56:20.055725       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:04:13.058038       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:11:44.060638       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:18:09.063625       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:27:33.066321       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:33:35.068669       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:39:40.071278       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:45:08.073594       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:54:55.076323       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:04:50.079296       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:14:12.082466       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:21:55.085285       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:31:43.087982       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:40:15.090289       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:47:37.092594       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:56:08.095083       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:03:34.097668       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:13:09.100620       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:21:02.102746       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:30:49.105306       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:40:09.108204       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:45:57.111446       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:54:31.113788       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:02:59.116735       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:12:31.119273       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:19:52.122071       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:29:14.124265       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:35:18.127574       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:42:08.129374       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:48:11.131754       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:54:43.134374       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:04:28.137033       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:12:29.139562       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:18:50.141696       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:26:50.144248       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:32:25.146684       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:42:06.149309       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:52:02.151572       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:59:58.154065       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:09:08.156325       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:16:55.159177       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:26:39.162276       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:33:26.164383       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:40:53.167410       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:48:40.170345       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:53:55.172629       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:01:06.174793       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:09:15.177854       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:16:34.180298       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:23:09.182713       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:28:11.184900       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:35:02.187067       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:43:53.189819       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:52:52.192431       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:57:55.195072       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:07:05.197860       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:12:14.200156       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:19:10.203680       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:28:10.206632       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:35:48.208923       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:41:13.211040       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:50:13.213889       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:56:45.216084       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:02:05.218621       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:09:54.221229       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:17:39.224193       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:22:45.227191       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:28:44.230239       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:34:28.232341       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:42:13.234975       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:48:34.237227       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:57:33.239862       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:03:35.241995       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:10:09.244505       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:18:15.247642       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:26:38.249967       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:33:52.252725       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:40:10.255515       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:47:46.258264       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:55:14.260540       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:02:02.263314       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:08:20.265968       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:16:23.268296       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:26:08.270480       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:34:08.272986       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:40:45.275749       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:49:13.278117       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:57:15.280080       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:02:49.282596       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:09:24.285563       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:15:02.288889       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:22:48.291680       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:32:01.294341       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:38:27.296394       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:44:19.300001       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:50:44.302332       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:56:59.304375       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:05:43.306947       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:13:38.309242       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:22:15.312452       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:28:59.314841       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:38:40.317870       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:44:27.320490       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:52:56.322578       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:02:43.324681       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:11:55.326762       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:20:20.329310       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:26:03.332278       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:35:38.334709       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:42:16.336667       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:48:11.339448       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:57:21.342223       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:05:00.345477       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:11:03.347516       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:18:44.350361       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:27:49.353339       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:35:36.356008       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:42:44.358608       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:51:17.361321       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:59:39.364465       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:05:39.367188       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:11:24.368994       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:18:24.371383       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:23:50.373928       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:33:29.375935       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:42:51.378391       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:51:41.381574       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:59:58.384095       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:07:50.386950       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:17:15.389825       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:25:47.392437       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:32:40.395278       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:38:51.398080       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:44:49.399848       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:52:46.402228       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:00:14.405044       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:10:04.407544       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:19:09.410025       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:25:07.412915       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:32:14.415455       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:38:39.417968       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:45:09.420462       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:54:35.422679       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:02:56.425564       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:10:12.428209       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:17:31.430939       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:24:41.433501       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:34:21.436731       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:40:02.439076       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:49:39.444471       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:59:15.447139       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:08:34.449521       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:16:54.451994       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:23:55.454598       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:31:09.457394       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:40:39.460206       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:49:58.462588       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:59:53.465576       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:05:19.467792       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:15:09.470551       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:22:55.473475       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:30:17.476141       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:37:20.479019       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:43:28.481904       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:49:07.484238       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:55:56.486016       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:05:45.488525       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:14:43.491319       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:21:42.494116       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:29:44.496659       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:38:45.499345       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:46:24.502182       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:54:30.504973       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:04:14.507454       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:10:37.510138       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:18:59.512979       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:28:22.515306       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:38:07.518167       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:43:14.520757       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:50:22.523374       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:58:19.526251       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:05:56.528244       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:15:28.531336       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:25:25.534139       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:34:26.536321       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:41:43.538490       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:51:40.541340       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:00:57.543966       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:08:04.546485       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:17:17.548562       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:25:24.550969       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:34:52.553832       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:42:42.555593       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:49:45.558369       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:55:16.560438       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:00:25.562811       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:06:38.564866       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:14:40.567222       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:22:14.569353       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:27:19.572141       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:33:57.574751       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:39:29.577301       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:48:14.580497       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:55:27.582467       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:05:17.584723       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:10:22.587643       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:18:34.590744       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:26:26.593501       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:33:23.595736       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:40:53.598521       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:48:08.601090       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:53:47.603525       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:02:59.606845       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:09:21.609133       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:14:50.611190       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:24:38.613297       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:32:36.616128       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:42:10.619009       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:47:55.621780       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:54:37.623850       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:02:45.626261       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:07:50.628457       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:14:53.630942       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:21:36.633512       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:28:09.635784       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:36:44.638394       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:42:45.641582       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:52:29.644478       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:57:52.646485       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:05:53.649528       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:13:36.652614       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:19:27.654595       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:26:04.657255       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:34:25.659688       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:43:37.662125       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:50:28.664156       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:55:54.666919       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:04:32.669344       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:10:00.672364       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:15:48.674661       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:22:34.677038       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:30:04.679605       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:40:01.682504       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:46:18.684345       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:52:43.687017       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:01:35.691616       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:10:29.694027       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:16:24.695777       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:21:55.698669       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:29:45.700781       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:38:48.703229       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:47:59.706579       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:53:15.709333       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:00:32.711380       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:07:25.713775       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:12:59.715762       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:18:13.717828       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:23:13.719669       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:31:08.722007       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:39:58.724660       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:46:11.727704       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:52:53.729880       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:01:13.732073       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:08:47.734707       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:18:10.737546       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:23:43.740470       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:29:06.743100       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:38:31.745470       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:47:35.748429       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:53:40.751244       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:02:44.753875       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:09:03.756439       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:18:13.758975       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:27:49.761672       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:35:44.764766       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:44:21.767723       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:49:21.770241       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:57:51.773335       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:03:50.776019       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:09:32.778664       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:19:02.781350       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:25:53.785797       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:35:19.788814       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:43:04.791919       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:51:19.794141       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:59:02.796552       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:04:52.798831       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:11:59.801915       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:21:48.804494       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:30:14.806862       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:38:58.809799       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:46:40.812592       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:56:11.815182       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:05:54.818371       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:13:41.820768       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:19:20.823540       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:28:40.826531       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:38:34.829212       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:45:54.832432       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:55:09.834843       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:00:49.837529       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:09:13.840567       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:18:30.843950       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:23:34.846283       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:31:26.848282       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:38:33.850517       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:44:41.853521       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:50:44.856177       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:59:35.858906       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:07:14.861305       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:15:14.863573       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:23:43.866872       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:31:01.868956       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:39:03.871604       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:47:18.873842       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:55:21.876946       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:02:36.879693       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:07:56.881519       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:14:02.884667       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:21:11.887300       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:30:40.890574       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:39:21.892877       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:47:34.895432       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:55:56.898123       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:00:59.900861       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:08:03.903371       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:16:44.905239       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:26:29.908067       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:36:28.909912       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:45:48.912453       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:51:39.914818       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:57:53.917426       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:04:31.919728       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:09:50.922627       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:19:16.925776       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:28:13.928009       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:33:54.930523       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:39:46.933504       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:48:00.935607       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:54:28.937954       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:03:21.940116       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:08:25.942387       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:14:22.944511       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:20:14.946982       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:28:13.949134       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:36:10.951263       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:45:47.954039       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:55:46.956655       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:01:22.958901       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:10:28.962117       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:18:40.964078       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:23:51.966832       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:31:32.969772       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:39:41.972272       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:48:02.975307       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:56:19.978546       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:05:11.980730       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:14:01.983107       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:23:48.986251       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:32:37.989107       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:40:32.991604       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:48:50.993337       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:55:58.995747       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:02:11.998339       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:09:21.000140       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:15:34.003007       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:23:55.005604       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:32:23.008152       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:42:09.010988       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:49:16.013728       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:54:48.016152       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:01:42.018710       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:10:26.021393       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:17:23.026436       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:25:21.029683       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:35:12.032517       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:40:33.034659       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:48:26.036948       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:56:37.039900       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:04:24.043128       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:11:11.045357       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:18:43.048159       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:27:27.051023       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:34:40.053278       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:42:11.055491       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:48:38.058499       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:56:42.061378       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:02:11.064299       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:08:22.067083       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:15:17.069009       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:20:55.071483       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:26:23.073592       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:34:34.076007       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:42:16.078640       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:49:29.081108       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:59:01.084160       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:04:59.086179       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:10:05.088845       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:17:15.091830       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:25:51.094762       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:31:11.097889       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:40:22.100029       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:49:38.102935       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:56:13.105079       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:03:21.107456       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:11:36.110516       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:20:33.112866       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:29:56.115583       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:35:33.117717       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:41:11.120285       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:49:57.122510       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:56:31.125112       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:02:20.127877       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:10:38.130856       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:20:29.133194       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:25:38.135775       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:35:10.138986       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:40:10.142118       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:48:24.145287       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:55:48.148203       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:04:01.150691       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:09:22.154030       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:16:39.157530       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:22:34.159973       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:27:43.162386       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:35:05.165286       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:43:14.168544       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:48:42.171301       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:58:04.174026       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:06:00.177193       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:15:49.180086       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:24:24.182883       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:32:15.185333       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:39:01.187249       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:44:43.190205       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:54:04.193440       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:02:44.196091       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:10:10.198557       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:17:04.201147       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:22:31.203541       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:28:35.206331       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:37:39.209131       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:42:44.211188       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:51:16.213466       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:57:45.215642       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:06:52.218936       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:15:37.221538       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:22:13.224051       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:30:57.226293       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:40:22.229032       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:48:03.231546       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:56:00.233592       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:04:49.235579       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:11:24.238197       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:20:07.241325       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:26:02.244272       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:35:41.247220       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:42:37.249960       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:51:47.252982       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:57:31.256449       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:06:54.259090       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:12:06.261672       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:20:27.264791       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:27:34.267267       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:37:23.269732       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:45:31.272833       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:53:18.275896       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:58:50.278740       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:06:31.281146       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:13:34.283699       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:20:57.285876       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:29:24.288468       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:35:31.291889       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:45:21.294990       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:51:55.298082       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:57:12.300848       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:04:43.303843       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:14:16.306229       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:21:44.308625       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:30:32.311377       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:36:35.313524       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:44:26.315866       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:49:35.318520       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:56:26.321470       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:03:27.323792       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:09:10.326484       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:18:40.329351       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:28:18.331990       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:38:05.334977       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:45:50.338129       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:53:08.341085       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:59:48.343782       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 06:08:29.347032       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 06:13:58.349325       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 06:23:38.352382       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 06:33:17.354953       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory

Pod: kube-flannel-ds-rfd8t (Namespace: kube-flannel)

Logs (filtered)

W0425 06:37:37.179997       1 reflector.go:462] pkg/subnet/kube/kube.go:470: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0425 06:37:48.183612       1 reflector.go:539] pkg/subnet/kube/kube.go:470: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90180": net/http: TLS handshake timeout
E0425 06:37:48.183693       1 reflector.go:147] pkg/subnet/kube/kube.go:470: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90180": net/http: TLS handshake timeout
W0425 06:38:01.346172       1 reflector.go:539] pkg/subnet/kube/kube.go:470: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90180": net/http: TLS handshake timeout
E0425 06:38:01.346231       1 reflector.go:147] pkg/subnet/kube/kube.go:470: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90180": net/http: TLS handshake timeout
W0425 06:38:15.041549       1 reflector.go:539] pkg/subnet/kube/kube.go:470: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90180": net/http: TLS handshake timeout
E0425 06:38:15.041605       1 reflector.go:147] pkg/subnet/kube/kube.go:470: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90180": net/http: TLS handshake timeout
W0425 06:38:33.389999       1 reflector.go:539] pkg/subnet/kube/kube.go:470: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90180": net/http: TLS handshake timeout
E0425 06:38:33.390055       1 reflector.go:147] pkg/subnet/kube/kube.go:470: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90180": net/http: TLS handshake timeout
W0425 06:39:03.288436       1 reflector.go:539] pkg/subnet/kube/kube.go:470: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90180": net/http: TLS handshake timeout
E0425 06:39:03.288506       1 reflector.go:147] pkg/subnet/kube/kube.go:470: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90180": net/http: TLS handshake timeout
W0425 06:39:53.240753       1 reflector.go:539] pkg/subnet/kube/kube.go:470: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90180": net/http: TLS handshake timeout
E0425 06:39:53.240797       1 reflector.go:147] pkg/subnet/kube/kube.go:470: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90180": net/http: TLS handshake timeout
W0425 06:40:38.540438       1 reflector.go:539] pkg/subnet/kube/kube.go:470: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90180": net/http: TLS handshake timeout
E0425 06:40:38.540524       1 reflector.go:147] pkg/subnet/kube/kube.go:470: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90180": net/http: TLS handshake timeout
W0425 06:41:30.608447       1 reflector.go:539] pkg/subnet/kube/kube.go:470: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90180": net/http: TLS handshake timeout
E0425 06:41:30.608499       1 reflector.go:147] pkg/subnet/kube/kube.go:470: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90180": net/http: TLS handshake timeout
W0425 06:42:25.184377       1 reflector.go:539] pkg/subnet/kube/kube.go:470: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90180": net/http: TLS handshake timeout
E0425 06:42:25.184449       1 reflector.go:147] pkg/subnet/kube/kube.go:470: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://10.96.0.1:443/api/v1/nodes?resourceVersion=90180": net/http: TLS handshake timeout
E0425 07:13:19.808677       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:22:26.811804       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:27:50.815526       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:35:59.818337       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:42:29.820875       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:52:24.827296       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:01:53.829986       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:08:41.834422       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:15:44.837079       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:25:00.840277       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:32:46.842371       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:37:54.845164       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:44:05.847705       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:53:16.850594       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:01:08.853690       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:06:43.860642       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:16:03.865462       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:21:13.869231       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:30:49.871616       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:37:37.874035       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:42:46.883225       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:51:14.885585       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:59:07.888781       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:07:13.892468       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:14:21.894759       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:21:45.897116       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:27:40.899606       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:34:23.903466       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:42:09.908022       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:47:58.911673       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:56:13.914339       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:02:44.918050       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:10:53.921612       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:17:17.923990       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:22:53.926156       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:28:07.929256       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:36:12.933034       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:45:02.936270       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:53:24.938844       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:01:33.942105       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:11:32.944716       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:17:00.947224       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:25:01.949669       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:30:21.952825       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:38:38.957563       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:46:07.961977       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:53:08.965944       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:00:01.970092       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:07:46.972645       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:15:45.975325       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:21:20.978226       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:27:40.981195       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:32:54.984344       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:41:22.986314       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:46:31.989048       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:54:47.991487       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:02:47.993889       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:11:04.996517       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:17:05.998775       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:23:01.001192       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:30:40.003090       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:40:13.005121       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:45:40.007776       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:52:04.010275       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:58:10.012174       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:06:06.013995       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:14:57.016503       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:23:33.018474       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:32:06.020800       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:40:17.023760       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:46:52.026253       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:54:33.028785       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:00:40.031664       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:06:35.034351       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:16:16.036970       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:24:34.040540       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:29:41.043407       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:37:25.046154       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:46:37.049317       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:52:42.051795       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:00:53.054975       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:10:34.059094       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:20:08.061195       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:25:34.063795       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:34:17.067153       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:43:06.070468       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:49:51.072972       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:58:03.076298       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:03:49.079123       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:12:07.082357       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:20:53.085320       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:29:38.087984       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:38:46.090682       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:47:09.093045       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:57:05.096431       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:05:30.098859       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:11:17.102059       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:20:29.105099       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:27:26.106991       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:34:23.109372       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:42:36.112061       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:48:16.114771       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:56:22.116715       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:06:10.118768       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:11:32.120974       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:18:57.122733       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:25:40.124662       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:30:56.126801       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:39:20.129136       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:46:40.131505       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:56:02.134009       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:03:39.136073       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:10:05.138073       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:17:46.140169       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:25:36.142098       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:32:26.144469       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:39:28.147324       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:45:26.150268       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:53:04.153122       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:58:36.156102       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:06:56.159990       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:14:46.163081       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:24:29.166382       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:30:54.168862       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:36:38.171091       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:43:57.174323       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:50:01.177332       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:56:21.179571       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:04:10.182559       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:10:38.185301       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:17:07.187814       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:25:10.190567       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:34:45.192959       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:44:18.195260       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:52:31.197939       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:59:55.200443       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:07:34.203304       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:16:38.205667       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:22:28.207991       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:31:03.210781       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:40:19.213850       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:49:30.216979       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:57:40.219323       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:05:49.222173       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:12:59.224753       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:19:09.227578       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:24:25.229879       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:31:06.233459       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:40:00.236010       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:46:38.238317       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:55:13.240600       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:03:20.243640       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:12:11.246036       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:19:28.248843       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:25:07.250699       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:30:28.252968       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:35:52.255071       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:42:21.258808       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:51:24.262007       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:57:53.264404       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:06:35.267002       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:15:45.270186       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:24:55.273254       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:33:48.275804       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:40:25.277748       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:46:15.280551       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:52:28.282612       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:57:41.284576       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:06:46.287136       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:12:40.290045       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:18:41.292216       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:24:18.295248       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:31:26.297791       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:36:44.300322       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:46:00.302264       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:55:32.304373       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:03:23.306962       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:12:41.309135       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:19:26.311818       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:26:12.314094       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:33:56.316681       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:39:04.319041       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:47:21.321371       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:57:17.324042       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:03:39.326619       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:12:38.328613       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:19:03.331693       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:24:46.333890       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:32:25.335785       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:39:38.338768       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:44:56.341041       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:51:59.343684       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:59:17.346015       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:05:14.348614       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:14:40.350983       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:21:53.353326       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:27:07.355693       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:33:42.358207       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:39:39.360513       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:46:23.363399       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:51:55.366590       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:01:01.369587       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:09:38.372303       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:19:15.375420       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:25:52.378449       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:34:30.381306       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:43:01.384236       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:48:15.386806       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:57:42.388894       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:06:51.391515       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:15:24.394584       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:23:16.397352       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:30:41.400255       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:37:44.402221       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:44:58.404812       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:53:46.406961       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:59:11.409223       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:04:24.411152       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:12:57.414108       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:19:05.416546       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:28:57.419226       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:34:59.422287       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:40:33.424880       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:45:41.427405       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:53:28.430140       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:03:11.432520       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:10:12.435497       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:16:53.438316       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:24:05.443621       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:33:43.448064       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:43:38.451192       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:50:59.454317       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:58:45.456445       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:06:43.459584       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:13:30.462481       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:22:05.464731       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:30:36.467213       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:37:40.469781       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:44:20.473095       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:50:17.474988       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:58:14.477122       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:07:45.479350       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:14:25.482175       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:20:31.484737       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:28:06.487519       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:34:40.489543       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:40:07.491338       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:48:54.493297       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:54:45.495629       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:03:02.498236       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:11:23.501015       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:17:33.503276       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:24:47.505633       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:32:47.508166       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:38:20.510273       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:46:42.512257       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:52:11.515012       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:00:51.517755       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:09:31.519704       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:19:02.522180       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:24:47.524087       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:31:37.526472       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:36:52.529362       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:46:07.532059       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:53:53.534168       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:00:11.537224       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:08:40.540133       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:15:15.545135       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:21:52.548137       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:28:42.550207       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:37:40.552308       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:43:41.554979       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:53:27.557012       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:01:31.559290       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:07:52.561859       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:13:35.564532       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:19:57.566686       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:28:40.569043       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:33:43.571492       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:41:47.573753       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:51:14.576345       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:56:51.579196       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:06:15.581452       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:15:45.584700       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:22:50.587055       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:28:47.589387       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:36:26.591970       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:45:30.594542       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:54:48.598430       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:02:52.601299       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:08:22.603607       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:14:32.605947       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:20:02.609151       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:25:46.612405       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:32:51.614835       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:40:51.618405       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:46:25.621330       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:53:49.624362       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:01:58.627604       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:08:58.630024       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:16:46.632650       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:22:20.635233       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:31:44.638310       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:38:35.642139       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:46:14.646149       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:54:19.648687       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:00:06.651558       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:05:08.653766       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:14:15.656829       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:21:38.660201       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:29:20.663286       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:37:20.665696       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:44:10.668350       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:51:14.671273       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:59:54.673323       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:09:21.676048       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:15:06.678812       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:20:31.681571       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:29:33.684083       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:35:03.686811       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:42:52.689302       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:49:18.691083       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:56:21.693179       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:02:26.695422       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:08:36.697495       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:14:34.699409       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:21:43.706531       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:28:57.708246       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:35:01.712197       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:42:10.714689       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:48:57.716730       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:57:16.719075       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:06:48.721619       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:14:55.724381       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:20:27.727519       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:26:27.729404       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:34:36.732230       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:43:40.734783       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:49:26.737237       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:56:47.739383       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:01:54.742814       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:10:46.746397       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:20:02.748645       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:28:56.750818       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:35:45.752574       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:41:01.754697       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:46:12.756849       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:51:47.759114       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:58:14.761060       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:04:16.763434       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:13:17.766049       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:19:06.768641       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:25:52.770635       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:31:27.773326       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:37:12.775060       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:44:15.777459       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:52:00.780058       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:01:57.783019       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:07:25.784896       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:17:17.788033       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:22:54.791009       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:29:28.793351       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:37:09.797113       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:46:51.799443       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:51:59.801459       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:57:36.804195       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:05:35.809649       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:11:26.811660       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:16:50.814876       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:25:03.817995       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:32:19.823644       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:41:37.826871       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:51:25.830689       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:58:50.833503       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:04:30.838053       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:09:42.843003       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:17:27.846100       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:25:38.848313       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:33:31.851016       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:38:33.853546       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:45:38.855835       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:51:46.857951       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:57:22.860092       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:02:25.862280       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:07:53.864739       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:12:56.867833       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:18:35.871472       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:24:26.873733       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:32:53.875808       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:41:53.878417       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:49:37.880935       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:58:15.883187       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:06:38.886001       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:16:10.888697       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:22:15.891262       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:31:26.894047       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:39:39.896056       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:47:36.898782       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:54:54.901533       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:01:49.904255       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:11:27.906980       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:21:25.909045       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:29:18.911063       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:35:00.913876       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:41:16.916846       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:49:03.919324       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:54:42.921835       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:02:12.924068       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:07:22.926045       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:16:53.928363       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:25:38.930856       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:33:37.933435       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:40:40.936478       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:50:21.938874       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:55:39.941803       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:03:09.944054       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:10:06.947045       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:15:57.949789       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:22:29.953042       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:31:32.956638       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:39:14.959856       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:48:15.962262       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:56:30.964309       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:02:59.966962       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:10:15.969406       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:16:04.973020       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:23:18.975648       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:31:32.977782       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:38:20.980590       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:47:14.983573       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:57:07.986516       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:06:30.989499       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:11:48.991619       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:18:51.994386       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:27:14.996593       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:37:12.999143       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:45:07.002012       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:54:45.004878       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:59:51.007135       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:09:49.010139       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:19:30.013274       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:29:16.016033       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:36:50.018724       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:44:27.021795       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:52:59.024333       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:00:05.026701       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:08:37.028981       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:18:24.031597       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:25:47.034098       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:33:42.036802       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:38:42.039303       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:47:37.042578       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:56:45.044674       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:04:45.047762       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:12:04.051005       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:18:53.054666       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:24:35.057591       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:33:12.060343       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:38:18.063369       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:44:53.065750       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:53:39.068564       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:03:04.071622       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:09:58.074260       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:18:55.076790       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:24:29.079521       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:32:02.081584       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:37:42.084699       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:44:48.088524       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:53:45.091418       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:00:51.093520       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:07:53.096107       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:12:55.099192       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:18:08.102003       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:25:28.105316       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:33:13.107961       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:40:00.110203       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:48:28.112419       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:56:02.115094       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:04:47.117321       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:12:20.120112       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:22:18.122700       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:32:04.125608       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:39:26.127708       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:45:57.129774       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:54:49.132083       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:02:36.134139       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:11:33.136584       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:17:49.138987       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:23:00.141469       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:29:14.144575       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:35:59.147174       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:42:39.149554       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:47:46.151682       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:53:38.154769       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:03:37.156997       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:10:33.159710       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:19:37.162430       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:28:25.166554       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:37:01.169310       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:42:27.172194       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:48:34.174360       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:56:23.176471       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:05:04.179714       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:14:41.181872       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:23:12.183872       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:30:51.186193       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:36:07.188522       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:45:04.191197       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:54:46.193876       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:03:52.197521       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:11:06.200112       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:20:26.202370       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:29:28.205306       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:37:25.207787       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:43:01.209880       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:52:44.212310       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:59:27.214500       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:07:57.216698       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:13:57.219283       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:23:00.221517       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:32:16.223988       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:40:04.227642       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:49:55.230403       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:58:18.232371       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:03:34.234649       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:11:32.236769       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:19:55.239484       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:26:36.242984       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:32:29.247780       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:41:35.250839       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:49:32.255994       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:57:37.259944       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:05:58.262584       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:12:54.266802       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:22:51.270259       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:28:34.272839       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:35:23.274876       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:45:14.277960       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:54:16.280469       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:59:49.282726       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:08:45.285580       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:18:42.288033       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:48.290462       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:14.293260       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:51.296145       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:48:10.298970       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:12.301066       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:38.303477       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:12:24.305251       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:21:31.307634       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:30:00.310224       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:37:30.312733       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:45:42.315490       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:55:04.317609       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:00:27.319507       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:06:49.321972       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:15:57.325106       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:25:16.327265       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:32:17.329859       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:42:05.332308       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:51:07.334977       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:59:22.337742       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:06:30.339992       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:11:41.342219       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:19:12.344871       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:25:23.347188       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:32:05.348995       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:40:47.351323       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:46:34.354085       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:54:58.357161       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:04:05.360079       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:09:32.362758       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:16:36.364553       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:25:19.369244       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:35:14.371465       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:41:08.374388       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:46:59.376005       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:55:30.378034       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:02:37.381182       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:11:52.384338       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:17:28.387099       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:24:22.389224       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:29:51.392683       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:38:45.396244       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:44:21.399181       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:50:07.401097       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:58:33.403965       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:05:27.406418       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:13:42.408880       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:23:21.412060       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:32:49.414096       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:38:50.416383       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:44:56.419241       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:54:28.421186       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:03:54.424556       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:13:31.426848       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:20:48.429593       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:27:23.435312       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:33:40.437470       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:39:10.439358       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:45:34.443880       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:53:42.445987       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:00:48.448805       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:05:54.451734       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:13:36.454826       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:22:09.458018       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:28:19.460413       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:33:24.462596       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:43:06.464637       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:48:42.467494       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:58:24.470307       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:07:06.473010       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:15:21.475766       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:24:56.478774       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:30:52.481078       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:37:41.482628       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:45:07.485074       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:52:32.487744       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:00:13.490133       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:05:49.492557       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:12:09.495472       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:19:37.497613       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:28:59.500552       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:35:44.503003       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:43:04.505682       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:52:42.507852       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:00:12.510492       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:07:25.512747       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:13:15.515374       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:21:18.518326       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:27:40.520476       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:36:09.522500       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:41:32.525270       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:48:10.527473       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:57:43.530475       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:04:51.533166       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:13:53.536002       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:21:05.538543       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:29:00.540740       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:38:26.543092       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:43:49.545584       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:49:03.548236       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:56:30.550736       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:05:51.553712       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:14:39.555819       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:20:57.558137       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:26:52.560519       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:34:38.563616       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:41:06.566721       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:49:31.569461       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:59:00.571528       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:08:19.574493       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:13:52.576819       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:19:46.579089       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:26:49.581666       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:31:51.584172       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:40:19.586216       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:46:43.588622       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:56:27.590366       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:02:28.592497       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:08:53.594434       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:16:18.596764       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:25:17.598690       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:34:45.601670       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:42:10.603590       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:50:12.606643       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:56:56.609206       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:03:25.611826       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:09:36.614444       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:18:13.617252       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:25:56.619310       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:32:17.622387       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:39:09.624895       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:45:42.627071       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:54:10.630021       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:03:02.632411       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:12:36.634677       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:18:07.637430       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:26:44.640046       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:35:32.642992       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:44:27.645180       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:53:29.648001       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:01:40.650410       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:08:58.652966       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:15:00.654855       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:23:43.657887       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:30:33.660496       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:36:50.662579       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:45:56.664668       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:55:39.667267       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:01:50.670063       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:07:20.672364       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:12:46.674588       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:18:30.677210       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:24:33.679173       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:30:16.681270       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:40:05.683521       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:45:22.686122       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:52:41.688806       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:00:08.691881       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:05:58.694006       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:14:09.697294       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:19:14.700357       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:28:11.702635       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:35:59.705236       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:45:43.707638       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:53:34.709775       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:59:41.712237       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:07:59.714861       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:17:14.718017       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:27:09.720377       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:34:29.723356       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:39:47.725570       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:47:43.728333       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:57:04.730488       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:03:12.733112       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:11:54.736245       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:17:17.739043       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:26:57.742238       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:34:30.745054       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:40:36.747749       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:49:48.750527       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:59:24.752643       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:05:08.754852       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:12:04.760786       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:17:25.763622       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:27:02.766252       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:35:12.768270       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:42:47.770613       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:49:31.773626       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:57:22.775880       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:06:27.778795       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:12:21.782025       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:21:27.783870       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:27:12.786525       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:35:42.789731       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:40:43.792339       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:49:51.794636       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:56:09.797623       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:05:36.799994       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:14:09.802505       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:22:55.805572       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:28:55.808092       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:35:18.810107       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:42:33.812240       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:47:50.815191       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:54:19.817571       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:03:55.819983       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:09:36.822495       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:16:56.824675       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:22:40.826767       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:28:21.829258       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:37:02.831803       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:42:22.834728       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:47:41.837091       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:57:37.840015       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:03:06.842479       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:08:15.845792       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:13:56.848373       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:20:31.850712       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:29:29.853877       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:37:12.856760       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:42:42.859423       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:51:33.862172       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:00:21.864358       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:06:42.866879       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:14:35.869692       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:21:33.872259       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:30:09.875092       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:39:53.877929       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:49:52.880730       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:57:31.883080       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:06:05.885404       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:14:58.888304       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:20:17.891062       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:29:22.893950       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:38:44.895951       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:48:04.898641       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:54:27.901464       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:03:35.904014       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:09:02.907270       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:15:32.909783       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:24:16.912593       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:31:51.915628       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:40:08.918403       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:45:11.920618       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:52:20.923703       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:00:07.925877       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:09:56.928566       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:19:16.931508       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:27:21.934058       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:33:04.936371       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:38:05.938849       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:47:09.941144       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:53:27.944047       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:03:04.946376       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:09:46.948498       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:18:32.951662       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:25:27.954137       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:33:43.957262       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:41:45.960430       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:50:41.963297       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:57:43.965664       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:04:23.968725       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:11:01.971719       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:18:32.974167       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:23:46.977019       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:29:59.979697       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:35:23.982471       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:44:01.984880       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:50:43.987379       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:58:59.990383       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:06:10.992492       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:12:32.994200       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:20:06.997100       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:25:20.999763       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:31:38.002129       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:41:21.004736       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:49:19.009009       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:57:27.012113       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:03:48.014604       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:11:41.017588       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:18:07.020080       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:26:08.022886       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:34:26.025075       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:42:16.027773       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:50:52.030665       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:59:53.032581       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:07:33.035537       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:12:45.038054       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:18:08.041075       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:25:07.044049       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:33:23.047395       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:41:21.049961       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:49:47.052452       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:57:03.055020       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:03:35.057568       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:12:17.060351       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:17:26.062616       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:23:03.064670       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:30:05.067316       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:36:22.070236       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:46:07.072983       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:55:58.076434       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:01:25.079149       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:06:46.081372       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:14:11.083945       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:20:52.085994       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:28:35.090973       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:36:27.096896       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:45:57.100209       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:54:13.103258       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:02:59.105656       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:12:53.108966       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:22:35.111549       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:28:16.113962       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:36:09.117091       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:41:33.119264       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:51:28.122345       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:58:38.125081       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:04:44.127061       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:10:24.129231       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:15:26.131933       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:21:06.134646       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:27:34.137038       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:35:06.139331       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:43:40.142247       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:49:11.145220       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:54:30.147077       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:59:31.149829       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:06:49.152486       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:14:36.154806       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:20:39.157037       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:30:19.159656       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:36:48.162641       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:44:54.165778       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:53:35.167821       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:02:02.170382       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:11:04.173151       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:20:06.176558       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:25:21.179424       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:31:33.181713       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:41:19.184344       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:46:21.187814       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:54:51.191555       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:01:07.194107       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:09:31.196483       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:18:23.198544       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:25:56.201195       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:31:33.203623       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:40:58.206582       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:47:54.208856       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:57:06.210875       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:03:43.213316       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:08:43.216128       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:16:13.218297       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:24:20.220974       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:30:25.223308       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:39:54.225946       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:46:14.228124       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:52:25.230365       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:01:09.232963       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:08:28.235170       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:15:04.237873       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:24:27.240743       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:34:00.243360       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:39:12.245847       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:47:09.248755       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:54:34.251656       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:02:32.253958       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:09:31.256379       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:17:24.258545       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:23:47.261410       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:31:12.264627       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:37:30.267499       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:44:47.269494       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:52:27.271628       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:59:43.274702       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:08:49.277597       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:15:03.280498       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:23:26.282695       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:32:46.285243       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:42:17.287648       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:47:20.289640       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:56:27.292059       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:03:51.294560       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:13:39.297669       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:22:42.300506       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:27:46.303689       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:36:45.306223       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:46:39.309042       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:55:01.311277       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:04:51.314081       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:09:53.316069       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:16:08.319083       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:23:41.321672       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:29:01.324363       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:35:39.327009       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:42:37.329081       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:50:15.331640       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:59:51.335033       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:08:02.337487       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:16:50.340359       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:25:21.343057       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:32:53.345127       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:41:17.347538       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:48:21.350392       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:56:02.353501       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:04:20.366183       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:14:05.369284       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:21:47.371695       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:28:28.373874       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:37:37.376667       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:45:37.378536       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:52:02.380793       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:00:56.383963       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:08:40.386040       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:14:26.388031       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:22:01.390795       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:28:42.393257       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:34:37.395976       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:42:40.398242       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:51:38.400782       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:00:01.403393       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:06:40.405950       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:13:34.408248       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:22:49.410585       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:29:38.412829       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:35:21.415464       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:40:35.418143       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:47:44.420494       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:53:23.423349       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:58:50.425741       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:07:10.429301       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:13:51.431845       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:19:20.434331       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:27:35.436756       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:35:43.439612       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:41:00.444651       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:48:54.447626       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:55:38.450323       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:04:13.451956       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:14:00.454488       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:19:35.457462       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:25:03.460362       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:32:25.463436       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:37:54.465873       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:44:30.468881       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:50:04.471749       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:55:15.474534       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:03:08.477181       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:10:04.479724       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:16:10.482389       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:23:22.484386       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:28:59.486948       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:36:36.489477       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:43:15.492128       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:51:48.494328       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:59:59.496685       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:07:13.499211       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:12:45.501397       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:22:07.504241       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:29:53.506672       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:37:46.509597       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:46:40.512073       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:52:23.515319       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:02:20.517451       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:11:47.519977       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:17:16.522981       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:22:21.525550       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:31:22.528155       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:36:53.530677       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:45:51.534013       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:50:58.536088       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:57:41.538789       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:05:41.541027       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:11:50.543503       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:21:02.545559       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:29:38.548169       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:35:08.551103       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:41:17.553981       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:49:27.556516       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:57:15.559051       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:05:07.561951       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:12:56.564151       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:20:55.566850       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:28:38.569555       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:36:43.572340       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:43:08.574604       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:52:33.576828       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:01:25.579374       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:06:46.581407       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:15:43.584144       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:23:13.587016       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:28:28.589302       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:37:46.591477       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:47:38.594327       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:56:32.596594       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:06:31.599128       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:16:18.601662       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:25:33.604544       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:32:20.606669       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:41:05.609333       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:48:57.611489       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:54:26.613203       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:59:36.615518       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:05:08.617878       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:11:07.620111       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:18:07.621660       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:23:51.624119       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:32:13.626328       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:38:46.629290       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:44:39.632607       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:49:40.635271       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:58:35.637561       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:08:16.640505       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:15:29.643136       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:23:27.645531       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:28:52.648400       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:37:49.651620       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:45:54.653801       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:54:32.655983       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:02:03.658122       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:10:50.660746       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:16:59.663596       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:22:42.666265       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:31:24.668413       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:37:47.671647       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:46:17.674062       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:52:49.676038       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:58:45.678984       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:08:16.682306       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:16:25.685315       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:22:32.687863       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:31:12.690992       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:39:56.693454       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:47:18.695884       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:56:24.697960       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:06:13.700991       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:13:38.703300       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:22:08.706297       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:29:20.708832       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:37:09.711797       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:46:18.714527       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:53:46.716772       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:59:22.719268       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:08:37.721816       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:17:24.724142       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:24:39.726433       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:31:05.728637       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:37:53.731323       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:45:46.734565       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:55:00.740604       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:01:11.742723       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:06:11.745687       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:14:01.749062       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:23:22.751973       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:31:05.754849       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:38:05.757802       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:43:16.760397       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:50:46.763422       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:55:53.766078       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:04:47.768630       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:09:48.771978       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:15:58.774491       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:23:17.777042       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:30:28.779586       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:37:42.781819       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:46:45.784057       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:55:16.786790       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:04:52.789048       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:12:27.792018       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:17:41.795244       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:27:24.798406       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:36:00.801603       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:43:27.803354       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:49:09.805506       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:55:53.808102       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:03:45.810738       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:10:14.813656       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:17:21.816048       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:24:48.819072       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:34:25.821569       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:40:13.824400       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:46:32.826561       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:56:09.829005       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:02:07.830964       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:08:25.833799       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:13:30.836568       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:21:18.839552       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:29:56.842667       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:37:52.844895       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:45:41.847685       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:54:56.850027       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:01:36.852461       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:11:33.854751       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:17:42.857755       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:27:28.860790       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:36:33.863765       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:43:15.866435       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:52:23.869199       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:01:07.871840       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:07:09.874500       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:12:49.876933       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:21:58.879386       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:29:02.882808       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:38:38.885670       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:47:21.887892       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:56:11.891929       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:01:54.894871       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:11:36.897625       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:19:07.900853       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:24:36.903970       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:31:00.907092       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:38:53.909131       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:48:24.911337       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:57:54.914412       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:04:10.917646       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:12:38.920765       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:19:13.923127       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:24:45.926018       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:30:01.928260       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:37:08.930707       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:46:12.933676       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:55:55.936059       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:05:43.939152       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:14:36.942039       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:23:25.944313       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:30:22.946505       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:37:24.949613       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:43:29.951789       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:50:59.954759       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:57:24.957020       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:04:00.959983       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:12:29.962205       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:20:44.964894       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:26:57.967649       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:35:41.970971       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:43:49.973186       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:49:30.976400       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:55:48.978673       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:05:31.980989       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:13:03.983371       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:21:22.986475       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:28:21.991195       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:35:35.993281       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:41:47.995500       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:50:55.997961       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:00:16.000700       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:09:18.002594       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:18:57.005444       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:26:55.007808       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:35:59.010106       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:42:57.012713       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:48:51.015187       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:54:23.017663       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:03:31.021406       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:10:33.023419       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:19:05.025781       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:24:57.027959       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:34:26.030734       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:42:00.033759       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:51:48.036165       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:00:24.038985       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:08:12.041352       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:16:46.044592       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:22:04.047312       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:30:51.050257       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:36:28.053674       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:46:04.056682       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:51:57.060479       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:57:50.063639       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:06:57.066510       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:12:33.068706       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:20:30.071644       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:28:37.074748       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:35:14.076893       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:44:58.080947       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:52:45.083609       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:02:21.085979       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:10:21.088665       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:15:47.091017       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:21:11.093598       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:28:42.095652       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:37:55.098251       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:45:05.100804       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:51:24.103683       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:57:44.106189       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:05:28.108804       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:13:42.112132       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:21:52.115217       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:30:46.117492       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:37:52.120324       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:47:24.123015       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:54:05.126213       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:03:07.128878       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:08:33.131801       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:15:48.134493       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:24:47.137735       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:34:03.139654       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:42:27.142317       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:51:21.145304       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:59:05.148300       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:05:56.151624       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:13:46.154414       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:22:24.157128       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:31:57.159567       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:40:44.162767       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:45:59.165056       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:53:17.167936       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:02:52.170495       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:10:23.172963       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:19:41.175541       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:28:05.178065       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:37:08.180229       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:45:25.183415       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:53:42.185772       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:59:51.188508       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:08:43.190861       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:16:28.193327       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:22:32.195342       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:29:20.198449       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:35:31.200515       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:44:13.202795       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:49:27.208287       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:55:59.211325       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:01:39.213562       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:11:26.215595       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:17:04.217728       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:23:59.220774       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:31:06.223156       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:38:52.225499       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:45:35.228126       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:52:12.230513       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:00:47.233163       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:09:47.235414       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:19:35.237360       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:25:17.240043       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:34:39.242818       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:40:31.245220       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:46:21.247858       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:52:42.250862       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:59:29.254041       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:05:42.256391       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:15:13.259083       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:24:17.261423       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:33:35.263690       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:41:17.266478       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:50:09.269523       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:55:23.271821       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:03:22.274889       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:10:34.278014       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:19:28.280442       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:29:24.282736       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:39:18.285642       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:48:50.288870       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:53:56.291514       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:03:41.294081       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:12:46.296413       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:21:20.299299       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:27:53.302259       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:37:25.305208       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:45:20.307737       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:50:44.310527       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:00:16.312830       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:09:19.315441       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:14:49.318084       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:23:27.320848       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:31:17.323706       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:37:17.325530       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:46:56.328125       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:53:35.330371       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:59:39.333341       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:08:26.335837       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:13:42.338850       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:21:02.342040       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:27:43.344880       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:32:47.347599       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:39:03.350053       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:47:49.352832       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:55:47.355678       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:04:56.358440       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:10:06.360610       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:18:19.363521       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:27:08.365956       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:34:04.368987       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:41:45.371470       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:48:52.374578       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:58:45.377438       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:03:45.380490       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:12:41.382642       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:20:36.385333       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:29:42.388428       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:37:56.391029       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:44:55.394006       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:52:56.396564       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:58:23.399666       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:05:54.402549       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:14:09.405618       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:23:04.408591       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:29:51.411273       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:39:03.414401       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:47:23.416575       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:54:50.419050       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:00:52.421322       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:07:51.423680       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:12:55.427146       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:19:15.430037       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:27:37.432863       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:34:56.435783       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:42:36.438489       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:48:59.443375       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:56:22.445347       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:02:05.448378       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:10:50.450705       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:18:42.453115       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:27:59.455424       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:34:28.458336       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:43:56.461593       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:53:15.464044       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:00:56.466618       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:07:10.469444       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:13:56.472563       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:22:17.474990       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:27:22.478305       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:35:51.480444       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:42:41.482686       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:47:45.484657       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:56:37.487232       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:04:23.490303       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:09:57.493327       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:18:23.495290       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:24:26.497262       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:31:19.499583       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:37:02.501801       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:43:47.504333       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:52:44.506860       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:02:05.510184       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:10:09.512977       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:17:47.515709       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:24:48.518389       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:32:45.520565       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:41:25.530767       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:47:00.532865       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:54:35.535061       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:01:49.537633       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:08:33.540111       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:18:10.542368       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:26:07.545321       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:33:09.548410       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:41:16.551150       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:49:28.553715       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:59:07.556407       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:06:52.559134       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:16:46.562068       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:25:41.564381       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:32:28.566807       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:40:54.569618       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:50:24.572392       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:56:44.574718       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:03:54.577133       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:12:13.579848       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:21:08.582652       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:29:32.585937       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:38:13.588525       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:43:56.591162       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:51:44.594225       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:01:34.597398       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:09:09.600330       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:18:19.602562       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:25:07.605414       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:31:03.607449       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:40:43.610230       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:47:36.612969       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:55:45.615737       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:01:15.618641       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:10:49.620891       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:20:18.623622       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:27:34.626751       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:34:13.628702       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:40:52.631301       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:45:52.633966       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:52:26.636496       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:57:53.639510       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:07:21.641798       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:13:25.644494       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:22:49.646427       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:29:03.649361       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:38:54.652070       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:45:07.654942       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:52:56.657285       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:58:44.659585       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:04:33.662329       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:09:36.664666       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:16:24.667075       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:23:31.669681       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:33:00.672165       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:42:29.675268       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:50:10.678080       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:00:05.681311       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:08:51.684050       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:14:45.686500       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:22:35.689025       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:31:41.692192       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:40:22.694834       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:48:30.696694       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:54:04.699541       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:00:13.702340       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:09:05.704545       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:17:18.706875       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:26:22.708854       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:31:32.711208       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:40:53.714026       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:48:59.716395       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:54:37.719243       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:01:16.721499       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:06:32.724358       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:13:44.726594       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:23:25.729750       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:29:08.733258       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:37:11.736219       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:46:52.738551       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:53:15.741129       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:58:21.743729       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:03:53.746237       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:09:39.748561       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:16:12.751431       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:22:12.753636       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:30:19.756546       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:36:02.759449       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:44:19.762756       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:52:13.764975       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:01:20.767731       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:06:47.770328       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:15:49.772545       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:22:51.775054       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:31:43.778211       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:39:49.780771       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:48:02.783042       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:56:00.785735       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:05:26.788053       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:13:25.790761       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:19:15.793358       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:25:09.795629       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:30:34.799547       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:35:49.802351       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:43:40.804820       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:49:01.808130       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:58:25.811533       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:06:28.813275       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:11:53.815529       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:19:11.818113       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:28:09.821400       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:35:38.823552       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:44:26.825873       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:53:10.828716       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:01:19.831707       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:10:43.833977       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:19:51.837003       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:25:41.839528       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:34:54.841464       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:40:46.843664       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:48:26.845622       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:54:08.848688       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:01:32.851530       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:09:10.853798       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:19:00.856579       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:27:48.859564       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:33:51.861590       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:39:17.864350       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:45:32.867260       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:53:32.870201       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:00:02.879555       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:06:29.882092       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:12:11.884485       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:17:32.886364       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:26:40.888544       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:32:10.891040       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:37:43.893453       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:46:32.895959       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:53:28.898444       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:02:29.900761       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:11:03.903505       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:18:09.905973       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:23:45.908651       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:30:08.911613       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:39:18.913643       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:46:31.916562       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:52:01.919491       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:01:41.922497       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:11:21.925435       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:20:51.928004       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:26:55.930460       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:35:46.932662       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:45:36.935862       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:55:24.938680       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:01:58.941189       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:08:42.944012       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:17:06.947206       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:25:32.949362       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:35:01.952502       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:40:33.955302       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:45:54.958041       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:51:10.960366       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:00:25.962881       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:10:04.965772       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:18:33.968121       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:28:02.970120       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:37:31.972840       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:46:37.976087       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:55:19.978508       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:03:43.980956       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:09:39.982834       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:17:52.985612       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:23:06.987815       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:31:23.990696       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:39:59.993258       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:46:19.995946       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:55:57.998830       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:03:49.001347       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:12:36.003762       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:20:03.006873       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:27:16.009622       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:33:02.011453       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:41:07.013688       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:46:12.015687       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:52:18.018211       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:57:22.020829       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:07:11.023209       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:12:47.025609       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:20:06.027672       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:25:41.030414       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:31:42.033357       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:39:52.036149       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:47:46.038756       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:56:52.041545       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:02:28.043864       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:11:15.046808       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:19:10.049210       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:25:21.051057       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:31:43.053769       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:39:44.056301       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:46:51.058592       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:52:00.060661       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:58:48.062638       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:07:43.064945       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:15:18.068155       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:22:06.071207       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:29:02.073472       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:35:26.076286       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:44:45.078950       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:53:46.081825       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:58:56.084622       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:07:14.087637       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:14:51.090883       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:22:54.093481       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:32:02.095657       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:39:16.098391       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:47:49.101134       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:57:29.103633       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:02:49.106482       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:08:59.108497       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:17:54.111171       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:26:17.113854       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:36:14.116461       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:42:46.119445       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:48:23.121390       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:54:12.123679       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:00:56.126846       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:08:00.129724       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:14:31.132097       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:20:59.134862       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:27:28.136867       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:35:13.140165       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:44:06.143174       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:51:14.145374       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:58:42.148611       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:04:40.150721       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:14:21.153763       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:20:42.156251       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:30:20.159111       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:37:36.161654       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:44:03.165030       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:49:40.167361       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:56:46.170560       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:05:05.172981       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:13:35.175476       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:22:03.177694       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:27:10.179958       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:32:53.182814       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:38:34.185817       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:44:28.188555       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:49:31.191563       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:56:17.193721       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:05:52.196561       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:14:00.198553       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:21:05.201114       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:26:44.203715       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:32:50.206775       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:38:00.208888       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:46:49.211470       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:53:47.214032       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:02:09.216420       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:10:40.218751       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:17:27.221585       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:23:35.224016       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:28:37.227431       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:34:53.230162       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:43:25.233070       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:48:32.235311       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:55:10.237754       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:01:39.240604       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:08:54.243615       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:15:09.246397       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:24:06.249547       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:29:52.252355       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:37:12.255385       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:45:46.258165       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:50:48.260419       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:56:11.262680       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:05:52.265075       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:13:35.268530       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:22:45.271095       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:31:48.273555       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:40:46.275595       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:47:18.277712       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:54:18.279963       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:02:25.282871       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:12:22.285901       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:20:39.288279       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:30:33.290685       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:40:18.293327       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:49:20.296247       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:58:48.298668       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:03:48.300743       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:09:14.303642       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:18:40.305964       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:26:54.308773       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:35:13.311523       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:42:18.314333       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:51:47.316595       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:01:40.319848       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:06:47.322783       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:12:03.325212       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:21:10.327796       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:29:31.330496       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:38:28.342207       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:48:04.344704       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:53:45.346606       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:00:08.348617       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:09:42.351328       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:15:10.354078       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:20:54.356954       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:26:40.359874       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:36:26.362040       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:45:52.364142       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:54:07.366336       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:00:05.369042       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:08:20.371202       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:15:34.374513       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:21:05.377110       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:28:12.379463       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:36:57.381959       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:43:43.385021       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:51:57.387690       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:00:47.390320       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:10:33.392699       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:18:25.395728       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:25:42.398480       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:31:15.401383       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:40:45.404530       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:48:59.406775       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:55:31.409680       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:02:13.412724       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:10:03.415478       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:16:42.418554       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:22:50.421740       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:29:14.424670       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:35:16.426760       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:43:57.429991       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:52:34.432355       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:01:35.434671       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:09:43.436667       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:15:44.439798       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:23:32.442664       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:30:24.444295       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:40:17.446165       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:46:00.448463       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:53:39.451311       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:02:07.453239       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:07:37.455578       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:14:14.458339       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:22:02.460684       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:30:32.463564       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:36:56.465721       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:42:35.468012       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:48:22.470056       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:57:24.472183       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:03:27.474650       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:08:55.476622       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:18:36.479892       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:25:23.482072       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:31:48.484846       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:39:13.487449       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:48:53.489700       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:57:01.492622       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:04:07.495288       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:11:52.498126       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:16:53.499967       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:24:06.502225       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:34:02.504512       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:42:11.506879       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:50:45.509759       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:57:36.513126       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:03:41.516133       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:09:50.518382       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:15:37.520535       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:20:45.523299       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:28:56.525770       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:35:24.528403       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:41:16.530546       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:47:45.532765       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:57:23.535030       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:06:12.538000       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:14:15.540779       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:23:49.543306       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:29:00.545453       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:36:43.547730       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:43:47.550144       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:49:43.552658       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:55:15.554724       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:02:55.556960       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:08:29.560000       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:13:30.563013       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:21:52.565522       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:28:15.568225       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:33:45.570987       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:42:46.573186       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:50:40.576254       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:56:42.578755       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:04:04.581431       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:10:23.584553       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:17:44.587788       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:27:38.590130       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:33:01.592361       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:42:36.595048       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:50:19.597985       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:57:09.600279       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:07:08.602980       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:14:22.606149       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:20:27.608784       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:28:12.611469       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:37:25.614231       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:44:38.616494       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:54:00.619368       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:01:45.621380       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:08:08.623473       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:14:58.625808       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:22:13.628204       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:30:36.630794       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:38:42.633199       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:46:52.635560       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:56:02.638210       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:03:43.641033       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:09:29.644213       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:14:51.647166       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:24:19.649766       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:33:09.652253       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:41:46.654519       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:51:15.657622       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:57:04.662344       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:03:30.664497       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:08:59.667262       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:16:45.669881       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:22:58.672183       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:29:35.674246       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:36:33.676494       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:42:50.679285       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:48:46.681627       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:57:56.684033       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:06:28.686488       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:12:10.689309       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:21:09.691710       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:30:39.694280       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:38:26.696797       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:43:48.698970       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:49:17.701500       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:58:36.704110       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:03:57.706748       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:10:55.709414       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:18:05.712096       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:25:10.714396       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:31:23.717050       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:38:56.719349       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:46:56.721733       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:52:21.723918       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:59:36.726701       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:09:02.729959       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:16:27.732419       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:24:31.734837       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:34:07.737422       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:40:13.739478       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:47:48.741931       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:55:31.744236       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:04:29.746619       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:09:50.749974       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:15:04.752416       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:20:32.754523       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:26:16.756950       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:36:08.759533       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:45:16.762171       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:50:45.765084       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:56:59.767791       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:03:49.781719       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:12:40.783689       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:19:20.786089       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:26:00.789230       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:34:05.791416       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:39:26.793996       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:46:24.796387       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:53:37.799172       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:59:23.801825       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:08:57.804333       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:14:40.806817       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:22:45.809568       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:29:50.811672       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:39:00.814172       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:44:35.817249       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:50:11.819237       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:58:29.821868       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:08:05.824555       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:17:14.827183       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:26:40.829020       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:33:40.830813       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:40:46.834336       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:47:35.836257       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:56:11.838626       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:02:17.840993       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:10:54.843694       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:18:16.846084       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:26:48.848190       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:34:53.850826       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:44:27.853259       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:50:52.855360       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:56:11.858562       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:02:05.861326       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:09:19.863769       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:15:05.866476       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:20:30.868856       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:26:46.871218       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:35:22.873855       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:44:12.876194       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:51:03.878875       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:56:44.881481       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:04:09.884285       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:12:27.886969       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:21:08.889428       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:26:41.892632       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:35:03.895121       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:43:42.897875       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:52:15.900208       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:58:31.902711       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:07:47.905834       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:15:28.908956       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:24:01.911445       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:29:52.914293       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:38:37.916575       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:47:17.918853       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:56:15.924127       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:02:32.927234       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:12:10.930088       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:19:32.933025       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:25:22.935033       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:35:01.936634       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:42:04.938545       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:47:48.940327       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:56:59.942928       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:04:00.945803       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:13:02.949003       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:20:38.951350       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:27:28.953653       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:33:49.956437       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:41:55.958136       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:47:50.961284       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:57:21.964067       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:03:23.966449       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:11:49.969095       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:20:36.971802       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:28:26.974207       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:34:24.977165       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:42:31.979291       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:49:01.981645       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:57:50.984136       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:07:33.987503       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:13:20.989705       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:21:21.991860       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:26:44.994974       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:34:57.998266       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:43:06.000988       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:51:44.003286       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:58:12.005404       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:05:25.007882       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:15:15.010525       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:23:18.012764       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:31:20.015024       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:39:07.017146       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:48:22.019848       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:53:40.022268       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:00:37.025040       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:09:47.027987       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:15:55.031099       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:24:50.034496       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:30:29.037019       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:38:45.040284       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:44:10.043006       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:49:24.045314       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:55:41.047841       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:03:41.050695       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:09:06.053410       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:14:57.056090       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:22:57.058388       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:32:17.060767       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:41:29.063442       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:47:47.066491       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:53:44.068718       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:03:17.070988       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:12:37.073148       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:18:15.075391       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:26:20.078069       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:32:49.080604       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:39:34.082490       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:49:15.084773       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:55:07.087067       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:02:59.089453       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:08:23.091337       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:14:44.093518       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:19:48.096297       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:25:14.098467       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:31:30.100872       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:36:30.103029       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:43:33.105686       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:49:45.107490       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:55:51.109741       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:02:50.112190       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:09:44.114816       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:16:38.117564       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:25:29.120219       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:33:15.122616       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:42:57.125613       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:51:02.128386       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:57:17.130695       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:03:04.134025       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:08:09.136655       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:16:18.139400       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:23:48.141232       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:30:30.143958       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:36:15.145869       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:42:22.148370       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:48:14.151628       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:54:58.153767       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:00:57.156388       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:09:09.158807       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:17:37.161818       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:22:44.165157       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:31:53.168241       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:37:04.171485       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:44:20.174052       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:53:50.176659       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:59:40.179025       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:07:23.181185       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:16:21.183280       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:21:25.186437       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:29:20.188973       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:37:01.192129       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:45:38.194768       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:55:27.197585       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:03:35.199960       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:09:37.202559       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:17:26.204709       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:23:48.207439       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:31:29.210081       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:39:31.212682       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:44:54.215298       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:52:52.217319       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:58:19.219598       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:04:14.222761       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:09:51.225419       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:19:10.228220       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:28:25.230662       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:37:31.233004       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:46:24.235254       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:55:33.237614       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:05:29.240662       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:10:51.243754       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:18:24.246649       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:28:09.249319       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:35:16.251299       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:41:55.253769       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:48:44.255557       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:56:40.258378       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:05:03.261350       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:15:01.264431       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:23:17.269529       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:30:03.271953       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:35:43.274463       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:43:25.277524       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:49:19.279841       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:57:49.282258       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:04:39.284655       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:10:30.286855       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:18:09.290042       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:28:05.293026       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:37:41.295427       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:44:44.297236       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:51:31.300038       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:57:56.302309       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:05:59.304851       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:12:36.307827       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:18:31.309899       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:27:27.312420       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:34:58.314790       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:43:35.317749       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:49:15.320107       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:57:44.322569       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:05:18.324820       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:12:45.327709       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:20:59.329794       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:27:23.332142       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:37:22.335016       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:45:58.337585       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:52:20.340370       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:57:25.342514       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:03:41.344675       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:10:02.346577       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:15:55.348756       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:23:59.351711       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:33:36.354646       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:38:49.356797       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:47:19.359823       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:53:26.363039       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:01:20.365532       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:07:24.368462       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:12:45.370250       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:21:38.372264       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:28:20.374998       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:33:36.377203       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:40:54.379587       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:49:00.382543       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:55:17.385280       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:04:48.387984       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:10:11.390814       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:16:33.392828       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:21:41.395574       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:26:54.397843       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:33:52.400990       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:39:57.404196       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:46:06.406850       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:52:33.409354       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:57:38.411998       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:05:05.414872       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:12:08.416700       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:19:47.418712       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:25:01.421099       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:33:56.424037       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:42:28.426405       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:51:22.428508       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:00:07.431174       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:06:55.433749       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:13:58.435746       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:20:10.438134       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:25:34.440506       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:31:32.444276       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:38:39.446820       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:43:57.449415       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:53:50.451381       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:00:43.454409       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:07:54.456610       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:16:30.459392       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:24:47.461984       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:29:49.464107       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:39:47.466744       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:46:04.469464       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:54:34.472208       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:00:38.475191       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:07:06.477737       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:15:21.479942       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:22:01.482535       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:27:51.485594       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:32:58.487819       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:41:17.490887       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:47:03.493367       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:53:42.495964       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:03:38.498782       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:11:58.500851       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:21:28.502684       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:27:37.504885       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:36:29.508048       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:41:35.510103       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:47:12.513102       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:52:40.515992       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:57:58.518219       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:06:11.519982       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:13:28.523277       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:22:03.525597       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:28:11.527649       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:36:58.530389       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:45:50.532985       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:54:51.535438       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:01:25.538044       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:07:20.540018       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:17:19.542245       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:22:37.544657       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:30:57.547680       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:40:12.549710       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:46:42.552330       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:54:59.554559       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:04:44.556732       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:14:03.559269       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:22:26.562531       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:29:30.565334       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:36:51.568836       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:45:06.571711       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:53:45.574566       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:02:35.577354       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:07:37.580135       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:16:28.582739       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:26:15.584647       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:31:36.586662       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:41:07.589432       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:49:16.592225       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:58:38.594990       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:05:33.597255       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:15:02.600419       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:22:13.603948       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:29:12.606269       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:39:07.609605       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:48:06.612430       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:57:41.615219       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:06:34.617147       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:14:11.619497       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:21:28.622352       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:31:03.625444       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:36:46.627858       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:41:51.630568       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:50:39.633047       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:00:27.635342       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:08:06.638228       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:16:14.640395       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:21:42.643291       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:27:32.646141       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:36:06.649136       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:42:53.651437       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:49:02.655093       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:57:43.658257       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:03:50.661410       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:09:11.663240       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:18:52.664815       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:23:58.667465       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:29:43.669428       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:38:31.671104       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:45:36.673609       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:51:22.676005       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:58:23.678447       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:07:29.681432       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:12:53.683380       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:22:25.685429       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:28:30.688251       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:33:32.690846       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:42:19.693378       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:49:03.695608       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:57:05.698503       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:05:50.701404       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:12:12.704315       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:21:30.707258       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:26:47.710191       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:36:46.712717       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:44:59.715661       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:50:05.718864       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:59:37.722463       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:07:57.725337       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:15:12.727977       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:24:55.730235       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:31:23.733379       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:37:23.735251       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:44:34.737809       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:50:57.740446       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:57:05.743321       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:06:59.746176       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:15:13.748706       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:20:24.751281       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:28:39.754200       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:34:33.757187       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:42:33.759887       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:51:53.762487       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:56:59.765029       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:02:57.767644       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:10:31.770483       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:19:40.773102       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:29:11.775565       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:34:27.777604       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:41:02.780164       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:46:36.782287       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:51:58.785012       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:59:09.787489       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:05:53.790171       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:14:40.794220       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:24:34.796611       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:30:07.799604       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:39:25.801628       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:45:59.804115       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:54:47.806789       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:02:42.808849       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:09:22.811203       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:16:41.814220       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:24:57.817073       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:31:20.819326       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:40:30.821757       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:47:25.824299       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:55:41.826490       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:02:07.828823       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:07:19.831756       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:16:08.834420       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:22:27.836943       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:31:34.839657       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:37:40.842325       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:46:34.845382       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:53:47.847568       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:03:03.850768       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:08:14.853457       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:14:48.856256       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:24:03.858998       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:33:45.861563       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:42:35.863942       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:48:18.866833       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:57:54.869297       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:06:56.871376       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:13:35.874015       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:18:55.876434       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:26:33.879280       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:33:57.881479       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:42:01.884518       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:49:18.886659       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:54:34.888832       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:01:40.892101       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:07:45.894389       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:15:16.897283       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:20:48.899413       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:29:41.902072       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:37:12.905092       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:43:17.907070       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:50:05.909136       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:57:56.911527       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:05:16.914086       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:15:11.916854       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:25:00.919596       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:30:18.922520       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:39:23.925287       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:47:17.927936       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:54:02.930677       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:03:48.933486       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:12:05.935564       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:17:45.937837       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:24:04.940898       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:30:25.943688       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:35:44.946117       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:44:46.948155       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:49:48.951040       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:57:33.953748       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:04:23.956010       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:10:25.958590       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:17:29.961800       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:26:12.964072       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:34:05.967323       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:41:32.970260       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:46:34.972860       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:54:50.975383       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:01:24.978444       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:10:28.981345       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:16:15.983596       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:22:14.986110       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:30:47.988716       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:40:35.990832       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:48:53.994456       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:57:16.996824       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:04:44.999853       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:13:20.002710       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:22:57.005614       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:28:31.007737       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:37:55.009748       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:43:21.011882       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:51:45.013612       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:00:42.015847       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:06:53.018743       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:15:55.021380       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:25:44.024398       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:32:44.027012       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:42:43.029641       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:51:25.032689       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:57:13.035109       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:05:43.038074       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:15:00.040596       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:21:12.043052       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:28:33.045137       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:34:28.047708       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:40:12.050544       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:50:03.052848       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:58:00.055981       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:04:28.058967       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:11:57.062177       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:17:36.065203       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:24:18.067325       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:30:16.070448       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:38:22.072804       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:45:50.075625       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:51:19.077571       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:00:44.080441       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:08:19.083030       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:16:37.085743       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:24:53.088637       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:33:50.091323       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:42:59.093567       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:52:06.095994       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:58:28.099064       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:03:56.102090       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:10:50.104222       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:17:23.107265       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:23:04.110326       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:31:03.112629       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:38:15.114714       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:47:37.117036       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:54:09.120004       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:59:24.121808       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:06:19.124313       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:13:26.126871       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:20:55.129195       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:26:49.131069       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:35:35.133503       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:43:30.136360       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:53:16.138717       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:00:51.141305       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:07:36.143957       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:14:54.146937       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:24:49.150101       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:33:24.152972       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:41:58.155730       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:49:01.158261       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:56:27.161310       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:03:23.163813       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:12:40.166756       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:21:46.169873       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:29:45.172429       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:37:44.174964       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:44:28.176889       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:51:08.179703       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:59:56.182017       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:07:32.184610       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:13:49.187822       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:21:24.190470       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:27:20.193636       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:35:24.196255       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:43:40.198748       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:49:15.201046       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:58:38.203836       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:04:49.206233       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:14:44.208554       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:21:28.211777       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:29:56.214159       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:36:09.216731       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:42:46.218749       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:51:27.221322       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:56:31.223177       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:05:28.225029       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:14:10.227317       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:21:34.229639       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:26:44.232767       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:34:21.234864       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:44:14.236877       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:53:39.238738       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:01:45.241523       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:07:09.244386       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:13:33.246650       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:22:18.249422       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:29:50.253124       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:37:45.255788       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:44:36.258693       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:54:29.261782       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:01:02.263722       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:07:36.266162       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:16:51.269098       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:25:12.271833       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:34:06.274743       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:40:20.277974       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:46:47.282484       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:52:18.284721       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:01:26.288006       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:11:16.290628       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:18:39.293035       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:25:03.295224       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:31:59.297282       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:39:51.299738       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:47:54.301837       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:55:58.304957       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:02:29.307986       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:12:17.310995       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:20:34.314309       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:27:11.317376       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:35:54.320090       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:44:31.322089       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:49:44.324164       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:57:04.326708       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:02:12.329846       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:09:01.332234       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:17:50.334145       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:26:31.336970       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:33:10.339550       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:40:25.342003       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:46:14.344688       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:54:51.347630       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:01:49.350835       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:11:19.353323       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:17:27.356038       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:26:37.359033       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:34:07.362068       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:41:20.364292       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:46:27.367187       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:53:18.369549       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:02:40.371486       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:10:09.374433       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:19:30.376797       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:29:28.380119       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:34:49.382047       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:40:57.384305       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:49:45.386551       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:56:05.389221       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:04:06.392084       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:12:37.395055       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:18:55.397715       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:27:49.400702       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:33:02.403403       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:42:24.406026       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:47:32.407839       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:56:01.410490       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:03:34.412586       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:12:38.415151       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:18:28.417731       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:27:08.419865       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:34:40.422816       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:41:52.424756       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:50:47.427258       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:56:42.430340       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:02:52.433566       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:10:29.436367       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:16:11.438881       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:21:48.442590       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:29:55.445138       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:36:50.447125       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:43:05.449882       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:50:34.452654       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:59:03.466899       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 06:06:39.469314       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 06:15:08.471832       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 06:20:23.474232       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 06:29:27.476746       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 06:35:38.479174       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory

Pod: coredns-76f75df574-fcnqj (Namespace: kube-system)

Logs (filtered)

[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.1.239:52017->8.8.8.8:53: i/o timeout
[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.1.239:34928->8.8.8.8:53: i/o timeout
[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.1.239:60650->8.8.8.8:53: i/o timeout
[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.1.239:59513->8.8.8.8:53: i/o timeout

Pod: coredns-76f75df574-mmccd (Namespace: kube-system)

Logs (filtered)

[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.2.198:42453->8.8.8.8:53: i/o timeout
[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.2.198:51932->8.8.8.8:53: i/o timeout
[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.2.198:50896->8.8.8.8:53: i/o timeout
[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.2.198:52759->8.8.8.8:53: i/o timeout
[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.2.198:38892->8.8.8.8:53: i/o timeout
[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.2.198:54437->8.8.8.8:53: i/o timeout
[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.2.198:40837->8.8.8.8:53: i/o timeout
[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.2.198:45915->8.8.8.8:53: i/o timeout
[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.2.198:59168->8.8.8.8:53: i/o timeout
[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.2.198:57768->8.8.8.8:53: i/o timeout
[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.2.198:38889->8.8.8.8:53: i/o timeout
[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.2.198:41296->8.8.8.8:53: i/o timeout
[ERROR] plugin/errors: 2 metadata-grpc-service. AAAA: read udp 10.244.2.198:60368->8.8.8.8:53: i/o timeout

Pod: etcd-k8s-kubeflow-master01 (Namespace: kube-system)

Logs (filtered)

{"level":"warn","ts":"2025-04-25T02:55:13.499535Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000001-0000000000004460.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T02:55:43.500159Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000001-0000000000004460.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T02:56:13.500695Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000001-0000000000004460.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T02:56:43.501254Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000001-0000000000004460.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T02:57:13.502354Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000001-0000000000004460.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T02:57:43.503754Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000001-0000000000004460.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T02:58:13.504925Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000001-0000000000004460.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T02:58:43.505961Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000001-0000000000004460.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T02:59:13.506472Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000001-0000000000004460.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T02:59:43.506898Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000001-0000000000004460.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:00:13.507874Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000001-0000000000004460.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:00:43.509116Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000001-0000000000004460.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:01:13.509718Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000001-0000000000004460.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:01:43.510755Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000001-0000000000004460.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:02:13.511806Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000001-0000000000004460.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:02:43.512517Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000001-0000000000004460.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:03:13.513585Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000001-0000000000004460.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:03:43.514461Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000001-0000000000004460.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:04:13.515678Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000001-0000000000004460.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:04:43.516125Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000001-0000000000004460.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:05:13.516894Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000001-0000000000004460.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:05:43.518713Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000001-0000000000004460.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:06:13.523556Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000001-0000000000004460.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:06:43.531723Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000001-0000000000004460.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:07:13.532579Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000001-0000000000004460.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:09:14.930312Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"766.40008ms","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure: >>","response":"size:18"}
{"level":"warn","ts":"2025-04-25T03:09:14.930546Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T03:09:14.162774Z","time spent":"767.72528ms","remote":"127.0.0.1:58560","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":476,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T03:14:14.163311Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"274.971926ms","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure: >>","response":"size:18"}
{"level":"warn","ts":"2025-04-25T03:20:13.594137Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000000d-00000000000074de.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:20:43.597613Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000000d-00000000000074de.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:21:13.598656Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000000d-00000000000074de.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:21:43.599906Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000000d-00000000000074de.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:22:13.600775Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000000d-00000000000074de.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:22:43.601245Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000000d-00000000000074de.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:23:13.601756Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000000d-00000000000074de.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:23:43.602841Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000000d-00000000000074de.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:24:13.603626Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000000d-00000000000074de.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:24:43.604749Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000000d-00000000000074de.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:25:13.605486Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000000d-00000000000074de.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:25:43.606462Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000000d-00000000000074de.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:26:13.609623Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000000d-00000000000074de.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:26:43.610671Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000000d-00000000000074de.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:27:13.611184Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000000d-00000000000074de.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:27:43.612061Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000000d-00000000000074de.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:28:13.613195Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000000d-00000000000074de.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:28:43.614445Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000000d-00000000000074de.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:29:13.615464Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000000d-00000000000074de.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:29:14.714495Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"544.474645ms","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure: >>","response":"size:18"}
{"level":"warn","ts":"2025-04-25T03:29:14.714719Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T03:29:14.169542Z","time spent":"545.147309ms","remote":"127.0.0.1:58382","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":581,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T03:29:43.616592Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000000d-00000000000074de.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:30:13.617184Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000000d-00000000000074de.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:30:43.618214Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000000d-00000000000074de.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:31:13.619622Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000000d-00000000000074de.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:31:43.622576Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000000d-00000000000074de.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:32:13.623333Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000000d-00000000000074de.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:32:43.624433Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000000d-00000000000074de.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:33:13.625130Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000000d-00000000000074de.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:33:43.625875Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000000d-00000000000074de.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:34:13.626097Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000000d-00000000000074de.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:34:14.166167Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"148.061909ms","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure: >>","response":"size:18"}
{"level":"warn","ts":"2025-04-25T03:34:15.634474Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"1.464457662s","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure: >>","response":"size:18"}
{"level":"warn","ts":"2025-04-25T03:34:15.634655Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T03:34:14.169602Z","time spent":"1.465019228s","remote":"127.0.0.1:58560","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":503,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T03:34:15.642789Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T03:34:14.170017Z","time spent":"1.47274169s","remote":"127.0.0.1:58560","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":553,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T03:34:15.642875Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T03:34:14.170242Z","time spent":"1.472599511s","remote":"127.0.0.1:58560","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":547,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T03:34:15.643503Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T03:34:15.261080Z","time spent":"382.376121ms","remote":"127.0.0.1:39238","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":526025,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T03:34:43.626554Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000000d-00000000000074de.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:39:14.096333Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"141.389661ms","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure: >>","response":"size:18"}
{"level":"warn","ts":"2025-04-25T03:39:14.948428Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"799.397428ms","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure: >>","response":"size:18"}
{"level":"warn","ts":"2025-04-25T03:39:14.948762Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T03:39:14.148555Z","time spent":"800.149004ms","remote":"127.0.0.1:58560","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":685,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T03:39:14.951459Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T03:39:14.412418Z","time spent":"539.007175ms","remote":"127.0.0.1:58560","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":573,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T03:39:14.951622Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T03:39:14.391242Z","time spent":"560.349752ms","remote":"127.0.0.1:39238","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":526025,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T03:44:43.706704Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:45:13.709660Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:45:43.719629Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:46:13.723586Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:46:43.724361Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:47:13.724972Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:47:43.726118Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:48:13.727171Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:48:43.728231Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:49:13.728796Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:49:43.729314Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:50:13.730015Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:50:43.730639Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:51:13.731014Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:51:43.732206Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:52:13.732974Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:52:43.733951Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:53:13.735108Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:53:43.735888Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:54:13.736532Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:54:43.737344Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:55:13.738263Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:55:43.738983Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:56:13.739993Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:56:43.740757Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:57:13.741521Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:57:43.741700Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:58:13.742575Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:58:43.743644Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:59:13.744179Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T03:59:43.750366Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:00:13.750978Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:00:43.752270Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:01:13.755419Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:01:43.762444Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:02:13.763177Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:02:43.764079Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:03:13.764753Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:03:43.765309Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:04:13.766219Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:04:14.522694Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"430.477157ms","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure: >>","response":"size:18"}
{"level":"warn","ts":"2025-04-25T04:04:14.523075Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T04:04:14.090209Z","time spent":"432.776479ms","remote":"127.0.0.1:58382","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":581,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T04:04:43.766932Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000018-0000000000009b86.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:15:13.842338Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:15:43.842541Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:16:13.843601Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:16:43.844344Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:17:13.845220Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:17:43.846197Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:18:13.847244Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:18:43.847852Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:19:13.848859Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:19:43.859622Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:20:13.860881Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:20:43.861248Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:21:13.861832Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:21:43.862346Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:22:13.862782Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:22:43.863911Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:23:13.864619Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:23:43.865021Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:24:13.865701Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:24:14.487336Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"368.50361ms","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure: >>","response":"size:18"}
{"level":"warn","ts":"2025-04-25T04:24:14.487573Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T04:24:14.118086Z","time spent":"369.444173ms","remote":"127.0.0.1:58560","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":631,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T04:24:43.866128Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:25:13.866585Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:25:43.867874Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:26:13.868469Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:26:43.869157Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:27:13.870107Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:27:43.871196Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:28:13.871715Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:28:43.872611Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:29:13.873249Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:29:14.577984Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"621.922362ms","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure: >>","response":"size:18"}
{"level":"warn","ts":"2025-04-25T04:29:14.578335Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T04:29:13.955499Z","time spent":"622.791914ms","remote":"127.0.0.1:58560","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":686,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T04:29:14.584174Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T04:29:13.955752Z","time spent":"628.359633ms","remote":"127.0.0.1:58560","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":622,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T04:29:43.873870Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:30:13.874842Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:30:43.880697Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000025-000000000000c03c.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:34:14.379876Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"276.147355ms","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure: >>","response":"size:18"}
{"level":"warn","ts":"2025-04-25T04:44:43.946977Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000030-000000000000ea0a.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:45:13.949633Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000030-000000000000ea0a.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:45:43.955887Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000030-000000000000ea0a.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:46:13.956628Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000030-000000000000ea0a.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:46:43.958589Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000030-000000000000ea0a.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:47:13.965610Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000030-000000000000ea0a.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:47:43.979597Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000030-000000000000ea0a.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:48:13.980790Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000030-000000000000ea0a.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:48:43.981256Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000030-000000000000ea0a.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:49:13.981707Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000030-000000000000ea0a.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:49:43.982596Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000030-000000000000ea0a.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:50:13.983802Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000030-000000000000ea0a.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:50:43.990695Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000030-000000000000ea0a.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:51:13.994817Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000030-000000000000ea0a.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:51:43.995130Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000030-000000000000ea0a.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:52:13.996140Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000030-000000000000ea0a.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:52:43.997218Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000030-000000000000ea0a.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:53:13.998461Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000030-000000000000ea0a.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T04:54:14.750826Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"621.424558ms","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure: >>","response":"size:18"}
{"level":"warn","ts":"2025-04-25T04:54:14.751157Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T04:54:14.068871Z","time spent":"682.230885ms","remote":"127.0.0.1:58560","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":547,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T04:54:14.865588Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"114.598881ms","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure: >>","response":"size:18"}
{"level":"warn","ts":"2025-04-25T04:54:14.865859Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T04:54:14.068969Z","time spent":"796.842716ms","remote":"127.0.0.1:58560","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":549,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T04:54:14.865957Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T04:54:14.068930Z","time spent":"796.99283ms","remote":"127.0.0.1:58560","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":595,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T04:54:14.866180Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T04:54:14.069255Z","time spent":"796.891094ms","remote":"127.0.0.1:58560","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":549,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T05:05:44.077666Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000003a-0000000000010ede.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:06:14.078074Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000003a-0000000000010ede.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:06:44.078316Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000003a-0000000000010ede.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:07:14.080025Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000003a-0000000000010ede.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:07:44.081186Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000003a-0000000000010ede.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:08:14.081882Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000003a-0000000000010ede.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:08:44.082330Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000003a-0000000000010ede.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:09:14.082731Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000003a-0000000000010ede.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:09:44.086910Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000003a-0000000000010ede.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:10:14.087768Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000003a-0000000000010ede.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:10:44.089267Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000003a-0000000000010ede.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:11:14.090488Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000003a-0000000000010ede.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:11:44.091728Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000003a-0000000000010ede.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:12:14.092795Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000003a-0000000000010ede.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:12:44.093037Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000003a-0000000000010ede.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:13:14.094176Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000003a-0000000000010ede.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:13:44.095099Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000003a-0000000000010ede.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:14:14.096379Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000003a-0000000000010ede.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:14:44.096613Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000003a-0000000000010ede.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:15:14.097850Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000003a-0000000000010ede.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:15:44.098898Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000003a-0000000000010ede.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:16:14.099510Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000003a-0000000000010ede.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:16:44.100614Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000003a-0000000000010ede.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:17:14.101333Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000003a-0000000000010ede.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:17:44.102404Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000003a-0000000000010ede.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:18:14.102934Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000003a-0000000000010ede.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:18:44.103413Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000003a-0000000000010ede.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:19:14.103662Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000003a-0000000000010ede.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:19:44.103932Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000003a-0000000000010ede.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:20:14.104426Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000003a-0000000000010ede.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:20:44.106578Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/000000000000003a-0000000000010ede.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:31:14.150679Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:31:44.151634Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:32:14.151919Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:32:44.152898Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:33:14.154022Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:33:44.154799Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:34:14.155770Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:34:44.156415Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:35:14.164561Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:35:44.165569Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:36:14.166670Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:36:44.167132Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:37:14.167884Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:37:44.168614Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:38:14.169229Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:38:44.170364Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:39:14.171683Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:39:44.172179Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:40:14.187771Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:40:44.188316Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:41:14.188969Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:41:44.189575Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:42:14.197709Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:42:44.198551Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:43:14.199425Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:43:44.200002Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:44:14.201048Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:44:14.681159Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T05:44:14.368155Z","time spent":"312.897502ms","remote":"127.0.0.1:58560","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":573,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T05:44:44.203607Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:45:14.204714Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:45:44.205818Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:46:14.206108Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:46:44.206592Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:47:14.207685Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:47:44.208831Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:48:14.209430Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:48:44.210009Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:49:14.210235Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:49:16.133070Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"295.354589ms","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure: >>","response":"size:18"}
{"level":"warn","ts":"2025-04-25T05:49:16.133265Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T05:49:15.302957Z","time spent":"830.277745ms","remote":"127.0.0.1:39238","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":526025,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T05:49:16.135939Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T05:49:15.538426Z","time spent":"597.479496ms","remote":"127.0.0.1:58560","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":573,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T05:49:44.213807Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:50:14.214377Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:50:44.215035Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:51:14.216596Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:51:44.217337Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:52:14.217861Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:52:44.218925Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:53:14.219976Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:53:44.221069Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:54:14.221548Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:54:14.840661Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"777.571711ms","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure: >>","response":"size:18"}
{"level":"warn","ts":"2025-04-25T05:54:14.840866Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T05:54:14.062541Z","time spent":"778.292284ms","remote":"127.0.0.1:58560","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":685,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T05:54:44.222232Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:55:14.225603Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:55:44.226062Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:56:14.226650Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:56:44.227293Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:57:14.228151Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:57:44.228560Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:58:14.229303Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:58:44.230552Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:59:14.231583Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T05:59:44.232679Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:00:14.232917Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:00:44.234239Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:01:14.235325Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:01:44.236009Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:02:14.239223Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:02:44.239808Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:03:14.241278Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:03:44.242878Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:04:14.244217Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:04:16.458245Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"2.002736949s","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/health\" ","response":"","error":"context deadline exceeded"}
2025/04/25 06:04:16 WARNING: [core] [Server #6] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
{"level":"warn","ts":"2025-04-25T06:04:16.674208Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"2.58681292s","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure: >>","response":"size:18"}
{"level":"warn","ts":"2025-04-25T06:04:16.674487Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T06:04:14.076406Z","time spent":"2.598041887s","remote":"127.0.0.1:39238","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":526025,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T06:04:16.968212Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T06:04:14.989353Z","time spent":"1.978754133s","remote":"127.0.0.1:58560","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":573,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T06:04:16.968668Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T06:04:16.377270Z","time spent":"591.369401ms","remote":"127.0.0.1:58560","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":573,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T06:04:44.248590Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:05:14.251694Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:05:44.252304Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:06:14.257553Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:06:44.258822Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:07:14.259711Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:07:44.261003Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:08:14.263610Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:08:44.266279Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:09:14.268192Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:09:16.452199Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"2.007711416s","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/health\" ","response":"","error":"context deadline exceeded"}
2025/04/25 06:09:16 WARNING: [core] [Server #6] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
{"level":"warn","ts":"2025-04-25T06:09:16.848348Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"2.719008875s","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure: >>","response":"size:18"}
{"level":"warn","ts":"2025-04-25T06:09:16.848781Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T06:09:14.127015Z","time spent":"2.721721007s","remote":"127.0.0.1:58462","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":10557,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T06:09:18.136791Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"1.288070414s","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure: >>","response":"size:18"}
{"level":"warn","ts":"2025-04-25T06:09:18.138362Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T06:09:14.131654Z","time spent":"4.006667781s","remote":"127.0.0.1:58560","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":685,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T06:09:18.189742Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T06:09:15.326361Z","time spent":"2.863327294s","remote":"127.0.0.1:39238","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":526025,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T06:09:44.274436Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:10:14.740040Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:10:23.542426Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"175d966a86ff87c9","error":"etcdserver: request timed out"}
{"level":"warn","ts":"2025-04-25T06:10:23.625889Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"4.562904ms","request":"header: lease_revoke:","response":"size:28","error":"lease not found"}
{"level":"warn","ts":"2025-04-25T06:10:23.625998Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"175d966a86ff87c9","error":"lease not found"}
{"level":"warn","ts":"2025-04-25T06:10:46.471428Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:11:16.477832Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:11:46.478138Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000044-000000000001345b.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:14:16.445792Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"2.000352598s","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/health\" ","response":"","error":"context deadline exceeded"}
2025/04/25 06:14:16 WARNING: [core] [Server #6] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
{"level":"warn","ts":"2025-04-25T06:14:16.952122Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"2.563472098s","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure: >>","response":"size:18"}
{"level":"warn","ts":"2025-04-25T06:14:16.952380Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T06:14:14.385406Z","time spent":"2.56693723s","remote":"127.0.0.1:39238","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":526158,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T06:14:17.735475Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"783.118145ms","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure: >>","response":"size:18"}
{"level":"warn","ts":"2025-04-25T06:14:17.735707Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T06:14:14.389425Z","time spent":"3.346241874s","remote":"127.0.0.1:58560","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":504,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T06:17:46.548261Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:18:16.548606Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:18:46.549542Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:19:16.444269Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"2.000243178s","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/health\" ","response":"","error":"context canceled"}
2025/04/25 06:19:16 WARNING: [core] [Server #6] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
{"level":"warn","ts":"2025-04-25T06:19:16.485600Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"2.443240646s","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure: >>","response":"size:18"}
{"level":"warn","ts":"2025-04-25T06:19:16.485926Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T06:19:14.040588Z","time spent":"2.445277079s","remote":"127.0.0.1:39238","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":526158,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T06:19:16.550339Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:19:16.691966Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"206.162785ms","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure: >>","response":"size:18"}
{"level":"warn","ts":"2025-04-25T06:19:16.692256Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T06:19:15.316634Z","time spent":"1.375592504s","remote":"127.0.0.1:58560","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":686,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T06:19:16.692331Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T06:19:14.497780Z","time spent":"2.194519513s","remote":"127.0.0.1:58560","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":573,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T06:19:16.692524Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T06:19:14.826981Z","time spent":"1.865518076s","remote":"127.0.0.1:58560","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":573,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T06:19:16.692824Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T06:19:15.652601Z","time spent":"1.040196772s","remote":"127.0.0.1:58560","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":573,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T06:19:46.551423Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:20:16.552269Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:20:46.555661Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:21:16.556471Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:21:46.557632Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:22:16.558916Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:22:46.559460Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:23:16.560630Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:23:46.561822Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:24:16.435463Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"2.377658307s","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure: >>","response":"size:18"}
{"level":"warn","ts":"2025-04-25T06:24:16.435811Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T06:24:14.056310Z","time spent":"2.379464274s","remote":"127.0.0.1:58560","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":572,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T06:24:16.563121Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:24:17.342252Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"2.000196896s","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/health\" ","response":"","error":"context deadline exceeded"}
{"level":"warn","ts":"2025-04-25T06:24:17.444017Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"1.999925858s","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/health\" ","response":"","error":"context deadline exceeded"}
{"level":"warn","ts":"2025-04-25T06:24:17.479772Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"1.043925091s","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure: >>","response":"size:18"}
{"level":"warn","ts":"2025-04-25T06:24:17.480113Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T06:24:15.298707Z","time spent":"2.18136144s","remote":"127.0.0.1:39238","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":526025,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T06:24:46.564359Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:25:16.565591Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:25:46.566601Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:26:16.567506Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:26:46.568912Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:27:16.570748Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:27:46.571461Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:28:16.579606Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:28:46.589900Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:29:16.590642Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:29:16.799564Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"2.186934142s","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure: >>","response":"size:18"}
{"level":"warn","ts":"2025-04-25T06:29:16.799937Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T06:29:14.581662Z","time spent":"2.218228552s","remote":"127.0.0.1:39238","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":526025,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T06:29:46.591998Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:30:16.592763Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:30:46.600630Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:31:16.601548Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:31:46.603732Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:32:16.605138Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:32:46.610831Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:33:16.611908Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:33:46.614604Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:34:16.444696Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"2.000902302s","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/health\" ","response":"","error":"context canceled"}
2025/04/25 06:34:16 WARNING: [core] [Server #6] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
{"level":"warn","ts":"2025-04-25T06:34:16.615240Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:34:17.942981Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"1.120863478s","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure: >>","response":"size:18"}
{"level":"warn","ts":"2025-04-25T06:34:17.943351Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T06:34:14.610801Z","time spent":"3.332521406s","remote":"127.0.0.1:58560","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":573,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T06:34:17.951318Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T06:34:15.423874Z","time spent":"2.527405512s","remote":"127.0.0.1:39238","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":526025,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T06:34:17.951636Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T06:34:16.822775Z","time spent":"1.128833089s","remote":"127.0.0.1:39222","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":119,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T06:34:17.951940Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T06:34:17.434461Z","time spent":"517.448803ms","remote":"127.0.0.1:58560","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":573,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T06:34:46.617656Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:35:16.618003Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:35:46.618500Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:36:16.622621Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:36:46.624633Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:37:17.866544Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:37:20.435122Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"175d966a86ffb500","error":"etcdserver: request timed out"}
{"level":"warn","ts":"2025-04-25T06:37:38.218032Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"175d966a86ffb548","error":"etcdserver: request timed out"}
{"level":"warn","ts":"2025-04-25T06:37:41.745888Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"175d966a86ffb548","error":"etcdserver: request timed out"}
{"level":"warn","ts":"2025-04-25T06:38:02.081280Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:38:52.689032Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:40:27.282504Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"11.454872322s","expected-duration":"100ms","prefix":"","request":"header: lease_revoke:","response":"size:28","error":"lease not found"}
{"level":"warn","ts":"2025-04-25T06:41:54.762164Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:42:25.189463Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"3m53.351159967s","request":"header: lease_revoke:","response":"size:28","error":"lease not found"}
{"level":"warn","ts":"2025-04-25T06:42:26.477491Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"339.097309ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/minions/k8s-kubeflow-worker01\" ","response":"","error":"context canceled"}
2025/04/25 06:42:26 WARNING: [core] [Server #6] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
{"level":"warn","ts":"2025-04-25T06:42:27.703468Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"1.834650697s","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure: >>","response":"size:18"}
{"level":"warn","ts":"2025-04-25T06:42:27.703686Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T06:42:25.815024Z","time spent":"1.888625787s","remote":"127.0.0.1:39222","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":118,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T06:42:27.705903Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T06:42:25.903913Z","time spent":"1.801959183s","remote":"127.0.0.1:57558","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":686,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T06:42:27.706371Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T06:42:26.203009Z","time spent":"1.503321952s","remote":"127.0.0.1:57558","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":573,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T06:42:27.707370Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T06:42:26.243883Z","time spent":"1.463458923s","remote":"127.0.0.1:57486","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":5974,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T06:42:27.708233Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T06:42:26.476015Z","time spent":"1.232177286s","remote":"127.0.0.1:57684","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":526025,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T06:42:55.144531Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:43:25.145744Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:43:55.146252Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:44:25.147522Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:44:55.148011Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:45:25.148501Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:45:55.149129Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:46:25.149500Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:46:55.149819Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:47:25.151051Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:47:55.151696Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:48:25.152649Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:48:55.153111Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:49:25.154504Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:49:55.155716Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:50:25.156686Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:50:55.157368Z","caller":"fileutil/purge.go:80","msg":"failed to lock file","path":"/var/lib/etcd/member/wal/0000000000000061-0000000000015f42.wal","error":"fileutil: file already locked"}
{"level":"warn","ts":"2025-04-25T06:53:48.880009Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"2.286368286s","expected-duration":"100ms","prefix":"","request":"header: txn: success:> failure: >>","response":"size:18"}
{"level":"warn","ts":"2025-04-25T06:53:48.880348Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T06:53:46.593071Z","time spent":"2.287205811s","remote":"127.0.0.1:51896","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":476,"response count":0,"response size":40,"request content":"compare: success:> failure: >"}
{"level":"warn","ts":"2025-04-25T06:53:48.956456Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T06:53:46.831488Z","time spent":"2.124870821s","remote":"127.0.0.1:51718","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":881,"response count":0,"response size":40,"request content":"compare: success:> failure:<>"}
{"level":"warn","ts":"2025-04-25T06:53:48.966733Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-04-25T06:53:47.912857Z","time spent":"1.053834044s","remote":"127.0.0.1:55612","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":950,"response count":0,"response size":40,"request content":"compare: success:> failure:<>"}

Pod: kube-apiserver-k8s-kubeflow-master01 (Namespace: kube-system)

Describe (filtered)

      Reason:       Error

Logs (filtered)

E0425 06:43:48.396521       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:43:49.419883       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:43:50.471678       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:43:51.503168       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:43:52.611564       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:43:53.697640       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:43:54.781004       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:43:55.872745       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:43:56.972534       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:43:58.049771       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:43:59.166171       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:44:00.280361       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:44:01.439045       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:44:02.602512       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:44:03.874643       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:44:05.040155       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:44:06.357592       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:44:07.687816       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:44:08.976272       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:44:10.269138       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:44:11.481865       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:44:12.725118       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:44:13.979185       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:44:15.278903       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:44:16.554628       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:44:17.848007       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:44:19.338159       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:44:20.609975       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:44:21.933491       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:44:23.208545       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:44:24.574342       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:44:25.879511       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:44:27.206195       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:44:28.841276       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
E0425 06:44:30.098316       1 crd_finalizer.go:299] inferenceservices.serving.kserve.io failed with: unable to find a custom resource client for inferenceservices.serving.kserve.io: unable to load root certificates: unable to parse bytes as PEM block
W0425 06:44:39.431793       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:39.687653       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:39.943779       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:40.135819       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:40.199609       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:40.263723       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:40.455656       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:40.711683       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:40.967756       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:41.159811       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:41.223749       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:41.287612       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:41.479904       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:41.735866       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:41.991719       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:42.183679       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:42.247736       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:42.311650       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:42.503677       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:42.759768       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:43.015799       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:43.207754       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:43.271636       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:43.335640       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:43.527645       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:43.783684       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:44.039737       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:44.232069       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:44.295715       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:44.359614       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:44.551843       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:44.807866       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:45.063829       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:45.256711       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:45.320074       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:45.383991       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:45.575809       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:45.831752       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:46.087747       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:46.279743       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:46.343673       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:46.407776       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:46.599698       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:46.855766       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:47.111662       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:47.303713       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:47.367647       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:47.431807       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:47.623835       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:47.879675       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:48.136142       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:48.199744       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:48.199869       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:48.200029       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:48.200197       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:48.200303       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:48.200433       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:48.200579       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:48.200630       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:48.200744       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:48.200903       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:48.199752       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:48.201001       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:48.200909       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:48.201164       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:48.201353       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:48.201539       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:48.328692       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:48.393857       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:48.457357       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:48.649426       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:48.904687       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:49.159706       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:49.354486       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:49.415639       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:49.481536       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:49.928679       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:49.928714       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:50.183696       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:50.247722       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:50.247798       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:50.247826       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:50.247846       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:50.247904       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:50.247722       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:50.247990       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:50.248018       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:50.247732       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:50.248152       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:50.375712       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:50.439658       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:50.503627       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:50.951830       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:51.207824       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:51.399761       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:51.463668       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:51.528673       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:51.975805       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:52.231809       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:52.423806       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:52.551715       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:52.551715       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:53.000795       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:53.255759       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:53.255760       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:53.319664       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:53.319692       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:53.319769       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:53.319876       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:53.319927       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:53.320045       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:53.320091       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:53.320203       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:53.320274       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:53.448705       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:53.575719       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:53.575875       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:54.023766       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:54.215739       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:54.215770       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:54.215739       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:54.215777       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:54.216664       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:54.216670       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:54.279772       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:54.471753       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:54.599700       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:54.599736       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:55.047669       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:55.303785       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:55.495654       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:55.623831       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:55.624001       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:56.071776       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:56.071900       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:56.327712       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:56.520668       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:56.647627       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:56.647666       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:57.095763       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:57.352819       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:57.543740       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:57.671893       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:57.671909       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:58.119823       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:58.311740       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:58.375659       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:58.375670       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:58.375758       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:58.375849       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:58.375886       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:58.375939       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:58.375758       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:58.376003       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:58.376093       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:44:58.376112       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:58.567861       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:58.695751       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:58.695824       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:59.143749       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:59.399861       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:59.591737       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:59.719812       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:44:59.719826       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:00.167729       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:00.424721       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:00.615837       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:00.743678       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:00.743689       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:01.191738       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:01.447715       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:01.639767       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:01.767877       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:01.767914       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:02.215727       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:02.471889       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:02.663767       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:02.791802       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:02.791799       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:03.240741       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:03.495711       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:03.687726       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:03.815604       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:03.815875       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:04.263619       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:04.519707       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:04.711837       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:04.839764       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:04.839780       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:05.223714       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:05.223715       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:05.223807       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:05.287661       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:05.287749       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:05.287924       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:05.287981       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:05.543762       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:05.735703       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:05.863939       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:05.863946       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:06.311811       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:06.568717       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:06.759670       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:06.887653       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:06.887689       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:07.335701       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:07.335750       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:07.335751       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:07.399628       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:07.399742       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:07.399786       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:07.399831       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:07.399956       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:07.399985       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:07.400028       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:07.399634       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:07.399633       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:07.591639       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:07.783701       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:07.911845       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:07.911985       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:08.359838       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:08.615686       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:08.807707       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:08.935652       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:08.935706       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:09.383717       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:09.639912       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:09.831831       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:09.959698       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:09.959906       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:10.407804       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:10.663815       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:10.855744       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:10.983776       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:10.983779       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:11.431773       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:11.687672       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:11.879741       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:12.007845       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:12.008086       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:12.455726       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:12.711853       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:12.903830       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:13.031679       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:13.031679       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:13.479792       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:13.735775       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:13.927710       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:14.055738       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:14.055738       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:14.503839       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:14.759797       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:14.951777       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:15.079761       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:15.079790       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:15.527709       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:15.783713       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:15.975723       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:16.103699       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:16.103701       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:16.551781       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:16.807824       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:16.999753       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:17.127768       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:17.127814       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:17.575686       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:17.831799       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:18.023762       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:18.151655       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:18.151844       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:18.599693       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:18.855788       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:19.047739       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:19.175714       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:19.176757       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:19.623769       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:19.879756       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:20.071719       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:20.199641       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:20.199807       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:20.647823       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:20.903858       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:21.095844       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:21.223751       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:21.223769       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:21.671774       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:21.927892       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:22.119715       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:22.248642       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:22.248753       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:22.695914       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:22.951707       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:23.143646       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:23.271726       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:23.271745       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:23.719745       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:23.975701       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:24.167808       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:24.295728       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:24.295864       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:24.359722       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:24.423661       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:24.423690       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:24.423768       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:24.423862       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:24.423911       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:24.423963       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:24.424055       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:24.424070       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:24.424175       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:24.743802       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:24.999832       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:25.191725       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:25.319779       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:25.319825       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:25.767780       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:26.023752       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:26.215777       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:26.279696       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:26.279713       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:26.279734       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:26.343788       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:26.343880       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:26.344022       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.110.185.215:443: connect: connection refused
W0425 06:45:26.344074       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:26.344018       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:26.791731       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:27.047769       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:27.239857       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:27.367792       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:27.367795       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:27.815849       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:28.071871       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:28.263771       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:28.391756       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:28.391858       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:28.839796       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:28.839796       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:29.095692       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:29.287716       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:29.415648       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:29.415676       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:29.863838       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:30.119923       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:30.311802       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:30.442621       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:30.442695       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:30.887764       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:31.143779       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:31.335674       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:31.463624       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:31.463625       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:31.911830       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:32.167763       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:32.359670       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:32.487633       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:32.487779       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:32.935714       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:33.191817       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:33.383839       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:33.511815       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:33.511815       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:33.959728       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:34.215779       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:34.407884       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:34.535684       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:34.536201       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:34.983914       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:35.239816       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:35.431693       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:35.559811       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:35.559795       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:36.007848       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:36.263703       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:36.455729       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:36.583629       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:36.583963       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:37.031843       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:37.287820       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:37.479804       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:37.607781       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:37.609001       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:38.055710       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:38.311833       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:38.503752       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:38.631690       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:38.631850       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:39.079827       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:39.335705       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:39.527663       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:39.655641       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:39.655724       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:40.103834       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:40.359789       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:40.551692       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:40.679754       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:40.679963       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:41.127770       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:41.383804       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:41.575742       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:41.703758       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:41.703877       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:42.151731       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:42.407791       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:42.599835       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:42.727736       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:42.727954       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:43.175793       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:43.431746       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:43.623775       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:43.751635       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:43.751639       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:44.199848       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:44.455638       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:44.647858       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:44.775767       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:44.776014       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:45.223755       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:45.479757       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:45.671668       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:45.799718       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:45.799818       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:46.247870       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:46.503693       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:46.695717       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:46.823677       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:46.823914       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:47.271688       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:47.527740       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:47.719711       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.103.16.215:443: connect: connection refused
W0425 06:45:47.868732       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": tls: failed to verify certificate: x509: certificate signed by unknown authority
W0425 06:45:47.868787       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": tls: failed to verify certificate: x509: certificate signed by unknown authority
W0425 06:45:47.988960       1 dispatcher.go:205] Failed calling webhook, failing open rev.validation.istio.io: failed calling webhook "rev.validation.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/validate?timeout=10s": tls: failed to verify certificate: x509: certificate signed by unknown authority
E0425 06:45:47.989092       1 dispatcher.go:213] failed calling webhook "rev.validation.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/validate?timeout=10s": tls: failed to verify certificate: x509: certificate signed by unknown authority
W0425 06:45:48.050261       1 dispatcher.go:225] rejected by webhook "rev.validation.istio.io": &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:"", RemainingItemCount:(*int64)(nil)}, Status:"Failure", Message:"admission webhook \"rev.validation.istio.io\" denied the request: configuration is invalid: \"internal.istio.io/webhook-always-reject\" annotation found, rejecting", Reason:"", Details:(*v1.StatusDetails)(nil), Code:400}}
W0428 02:25:55.608258       1 dispatcher.go:205] Failed calling webhook, failing open config.webhook.eventing.knative.dev: failed calling webhook "config.webhook.eventing.knative.dev": failed to call webhook: Post "https://eventing-webhook.knative-eventing.svc:443/config-validation?timeout=10s": service "eventing-webhook" not found
E0428 02:25:55.608324       1 dispatcher.go:213] failed calling webhook "config.webhook.eventing.knative.dev": failed to call webhook: Post "https://eventing-webhook.knative-eventing.svc:443/config-validation?timeout=10s": service "eventing-webhook" not found
W0428 02:25:55.621909       1 dispatcher.go:205] Failed calling webhook, failing open config.webhook.eventing.knative.dev: failed calling webhook "config.webhook.eventing.knative.dev": failed to call webhook: Post "https://eventing-webhook.knative-eventing.svc:443/config-validation?timeout=10s": service "eventing-webhook" not found
E0428 02:25:55.621959       1 dispatcher.go:213] failed calling webhook "config.webhook.eventing.knative.dev": failed to call webhook: Post "https://eventing-webhook.knative-eventing.svc:443/config-validation?timeout=10s": service "eventing-webhook" not found
W0428 02:25:55.623242       1 dispatcher.go:205] Failed calling webhook, failing open config.webhook.eventing.knative.dev: failed calling webhook "config.webhook.eventing.knative.dev": failed to call webhook: Post "https://eventing-webhook.knative-eventing.svc:443/config-validation?timeout=10s": service "eventing-webhook" not found
E0428 02:25:55.623279       1 dispatcher.go:213] failed calling webhook "config.webhook.eventing.knative.dev": failed to call webhook: Post "https://eventing-webhook.knative-eventing.svc:443/config-validation?timeout=10s": service "eventing-webhook" not found
W0428 02:25:55.869267       1 dispatcher.go:205] Failed calling webhook, failing open config.webhook.eventing.knative.dev: failed calling webhook "config.webhook.eventing.knative.dev": failed to call webhook: Post "https://eventing-webhook.knative-eventing.svc:443/config-validation?timeout=10s": service "eventing-webhook" not found
E0428 02:25:55.869315       1 dispatcher.go:213] failed calling webhook "config.webhook.eventing.knative.dev": failed to call webhook: Post "https://eventing-webhook.knative-eventing.svc:443/config-validation?timeout=10s": service "eventing-webhook" not found
W0428 02:25:55.869477       1 dispatcher.go:205] Failed calling webhook, failing open config.webhook.eventing.knative.dev: failed calling webhook "config.webhook.eventing.knative.dev": failed to call webhook: Post "https://eventing-webhook.knative-eventing.svc:443/config-validation?timeout=10s": service "eventing-webhook" not found
E0428 02:25:55.869514       1 dispatcher.go:213] failed calling webhook "config.webhook.eventing.knative.dev": failed to call webhook: Post "https://eventing-webhook.knative-eventing.svc:443/config-validation?timeout=10s": service "eventing-webhook" not found
W0428 02:25:55.870546       1 dispatcher.go:205] Failed calling webhook, failing open config.webhook.eventing.knative.dev: failed calling webhook "config.webhook.eventing.knative.dev": failed to call webhook: Post "https://eventing-webhook.knative-eventing.svc:443/config-validation?timeout=10s": service "eventing-webhook" not found
E0428 02:25:55.870578       1 dispatcher.go:213] failed calling webhook "config.webhook.eventing.knative.dev": failed to call webhook: Post "https://eventing-webhook.knative-eventing.svc:443/config-validation?timeout=10s": service "eventing-webhook" not found
W0428 04:11:21.031736       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:21.351686       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:21.479847       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:21.543690       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:21.671862       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:21.799919       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:22.055667       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:22.375828       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:22.503627       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:22.567629       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:22.695788       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:22.823772       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:23.079738       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:23.399695       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:23.527731       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:23.591838       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:23.719683       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:23.847755       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:24.103751       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:24.423803       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:24.552755       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:24.615645       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:24.743658       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:24.871686       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:25.127704       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:25.511683       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:25.575614       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:25.639708       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:25.767744       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:25.895663       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:26.151827       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:26.535820       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:26.599632       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:26.663795       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:26.791843       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:26.919801       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:27.175803       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:27.559683       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:27.623679       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:27.687683       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:27.815795       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:27.943741       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:28.199769       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:28.584684       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:28.647627       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:28.711699       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:28.839697       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:28.967746       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:29.223817       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:29.609045       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:29.671659       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:29.735689       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:29.863733       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:29.991636       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:30.119673       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:30.119766       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:30.119884       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:30.119967       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:30.120067       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:30.120206       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:30.120279       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:30.120380       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:30.120453       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:30.119673       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:30.120538       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:30.183623       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:30.183711       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:30.183808       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:30.183814       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:30.183910       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:30.247717       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:30.631907       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:30.695687       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:30.759731       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:30.888798       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:31.015810       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:31.527661       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:31.655706       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:31.719665       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:31.783616       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:31.911667       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:32.039769       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:32.167601       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:32.167852       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:32.167927       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:32.168015       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:32.167927       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:32.231710       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:32.231818       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:32.231837       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:32.231971       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:32.231994       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:32.679699       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:32.743694       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:32.807778       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:32.935832       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:33.063840       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:33.703701       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:33.767668       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:33.831810       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:34.023771       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:34.087723       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:34.727749       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:34.791740       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:34.855696       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:35.047695       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:35.111638       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:35.239656       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:35.239747       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:35.239656       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:35.239771       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:35.239904       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:35.239988       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:35.240063       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:35.239996       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:35.240087       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:35.240189       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:35.752664       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:35.815832       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:35.879709       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:36.071694       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:36.135670       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:36.135683       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:36.135747       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:36.135872       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:36.135998       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:36.136062       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:36.136066       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:36.775678       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:36.839685       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:36.903653       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:37.095789       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:37.159701       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:37.671720       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:37.799880       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:37.863668       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:37.927741       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:38.119795       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:38.183633       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:38.823699       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:38.887737       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:38.954833       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:39.143771       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:39.207750       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:39.847748       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:39.911799       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:39.975677       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:40.167744       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:40.231927       1 dispatcher.go:225] Failed calling webhook, failing closed namespace.sidecar-injector.istio.io: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:40.295718       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:40.295786       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:40.295803       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:40.295980       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:40.295988       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:40.296044       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:40.296154       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:40.296832       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:40.296916       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:40.296835       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:40.615728       1 dispatcher.go:205] Failed calling webhook, failing open rev.validation.istio.io: failed calling webhook "rev.validation.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/validate?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
E0428 04:11:40.615785       1 dispatcher.go:213] failed calling webhook "rev.validation.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/validate?timeout=10s": dial tcp 10.101.248.143:443: connect: connection refused
W0428 04:11:41.645096       1 dispatcher.go:225] rejected by webhook "rev.validation.istio.io": &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:"", RemainingItemCount:(*int64)(nil)}, Status:"Failure", Message:"admission webhook \"rev.validation.istio.io\" denied the request: configuration is invalid: \"internal.istio.io/webhook-always-reject\" annotation found, rejecting", Reason:"", Details:(*v1.StatusDetails)(nil), Code:400}}
W0428 04:11:47.207687       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:47.207775       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:47.207898       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:47.208064       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:47.208136       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:47.211044       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:49.319684       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:49.319773       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:49.319794       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:49.319907       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:49.319932       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:49.320013       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:49.320135       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:49.322006       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:49.322061       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:11:49.322239       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:12:06.343739       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:12:06.343786       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:12:06.343846       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:12:06.343979       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:12:06.343994       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:12:06.344162       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:12:06.344201       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:12:06.344264       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:12:06.344367       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:12:06.344439       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:12:08.327730       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:12:08.327793       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:12:08.327839       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:12:08.327730       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:12:08.327730       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:12:08.328040       1 dispatcher.go:217] Failed calling webhook, failing closed webhook.cert-manager.io: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": dial tcp 10.109.124.155:443: connect: connection refused
W0428 04:22:50.992752       1 dispatcher.go:225] rejected by webhook "rev.validation.istio.io": &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:"", RemainingItemCount:(*int64)(nil)}, Status:"Failure", Message:"admission webhook \"rev.validation.istio.io\" denied the request: configuration is invalid: \"internal.istio.io/webhook-always-reject\" annotation found, rejecting", Reason:"", Details:(*v1.StatusDetails)(nil), Code:400}}
W0428 05:53:00.999775       1 dispatcher.go:205] Failed calling webhook, failing open rev.validation.istio.io: failed calling webhook "rev.validation.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/validate?timeout=10s": dial tcp 10.96.59.161:443: connect: connection refused
E0428 05:53:00.999832       1 dispatcher.go:213] failed calling webhook "rev.validation.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/validate?timeout=10s": dial tcp 10.96.59.161:443: connect: connection refused
W0428 05:53:02.030076       1 dispatcher.go:225] rejected by webhook "rev.validation.istio.io": &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:"", RemainingItemCount:(*int64)(nil)}, Status:"Failure", Message:"admission webhook \"rev.validation.istio.io\" denied the request: configuration is invalid: \"internal.istio.io/webhook-always-reject\" annotation found, rejecting", Reason:"", Details:(*v1.StatusDetails)(nil), Code:400}}
W0428 06:45:47.975700       1 dispatcher.go:217] Failed calling webhook, failing closed clusterservingruntime.kserve-webhook-server.validator: failed calling webhook "clusterservingruntime.kserve-webhook-server.validator": failed to call webhook: Post "https://kserve-webhook-server-service.kubeflow.svc:443/validate-serving-kserve-io-v1alpha1-clusterservingruntime?timeout=10s": dial tcp 10.111.163.159:443: connect: connection refused
W0428 06:45:48.999798       1 dispatcher.go:217] Failed calling webhook, failing closed clusterservingruntime.kserve-webhook-server.validator: failed calling webhook "clusterservingruntime.kserve-webhook-server.validator": failed to call webhook: Post "https://kserve-webhook-server-service.kubeflow.svc:443/validate-serving-kserve-io-v1alpha1-clusterservingruntime?timeout=10s": dial tcp 10.111.163.159:443: connect: connection refused
W0428 06:45:50.023804       1 dispatcher.go:217] Failed calling webhook, failing closed clusterservingruntime.kserve-webhook-server.validator: failed calling webhook "clusterservingruntime.kserve-webhook-server.validator": failed to call webhook: Post "https://kserve-webhook-server-service.kubeflow.svc:443/validate-serving-kserve-io-v1alpha1-clusterservingruntime?timeout=10s": dial tcp 10.111.163.159:443: connect: connection refused
W0428 06:45:51.047740       1 dispatcher.go:217] Failed calling webhook, failing closed clusterservingruntime.kserve-webhook-server.validator: failed calling webhook "clusterservingruntime.kserve-webhook-server.validator": failed to call webhook: Post "https://kserve-webhook-server-service.kubeflow.svc:443/validate-serving-kserve-io-v1alpha1-clusterservingruntime?timeout=10s": dial tcp 10.111.163.159:443: connect: connection refused
W0428 06:45:52.071755       1 dispatcher.go:217] Failed calling webhook, failing closed clusterservingruntime.kserve-webhook-server.validator: failed calling webhook "clusterservingruntime.kserve-webhook-server.validator": failed to call webhook: Post "https://kserve-webhook-server-service.kubeflow.svc:443/validate-serving-kserve-io-v1alpha1-clusterservingruntime?timeout=10s": dial tcp 10.111.163.159:443: connect: connection refused
W0428 06:45:53.095615       1 dispatcher.go:217] Failed calling webhook, failing closed clusterservingruntime.kserve-webhook-server.validator: failed calling webhook "clusterservingruntime.kserve-webhook-server.validator": failed to call webhook: Post "https://kserve-webhook-server-service.kubeflow.svc:443/validate-serving-kserve-io-v1alpha1-clusterservingruntime?timeout=10s": dial tcp 10.111.163.159:443: connect: connection refused
W0428 06:45:54.119748       1 dispatcher.go:217] Failed calling webhook, failing closed clusterservingruntime.kserve-webhook-server.validator: failed calling webhook "clusterservingruntime.kserve-webhook-server.validator": failed to call webhook: Post "https://kserve-webhook-server-service.kubeflow.svc:443/validate-serving-kserve-io-v1alpha1-clusterservingruntime?timeout=10s": dial tcp 10.111.163.159:443: connect: connection refused
W0428 06:45:55.143772       1 dispatcher.go:217] Failed calling webhook, failing closed clusterservingruntime.kserve-webhook-server.validator: failed calling webhook "clusterservingruntime.kserve-webhook-server.validator": failed to call webhook: Post "https://kserve-webhook-server-service.kubeflow.svc:443/validate-serving-kserve-io-v1alpha1-clusterservingruntime?timeout=10s": dial tcp 10.111.163.159:443: connect: connection refused
W0428 06:45:56.168648       1 dispatcher.go:217] Failed calling webhook, failing closed clusterservingruntime.kserve-webhook-server.validator: failed calling webhook "clusterservingruntime.kserve-webhook-server.validator": failed to call webhook: Post "https://kserve-webhook-server-service.kubeflow.svc:443/validate-serving-kserve-io-v1alpha1-clusterservingruntime?timeout=10s": dial tcp 10.111.163.159:443: connect: connection refused

Pod: kube-controller-manager-k8s-kubeflow-master01 (Namespace: kube-system)

Describe (filtered)

      Reason:       Error

Logs (filtered)

W0508 05:52:47.497755       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:52:47.497812       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:52:51.516448       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:52:51.516515       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:52:57.700993       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:52:57.701037       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:52:57.901500       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:52:57.901554       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:52:58.122608       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:52:58.122608       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:52:58.122610       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:52:58.122610       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:52:58.122700       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:52:58.122717       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:52:58.122741       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:52:58.122773       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:52:58.122794       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:52:58.122814       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:52:58.122840       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:52:58.122859       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:52:58.122876       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:52:58.122894       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:52:58.122913       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 05:52:58.487582       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:52:58.487627       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:53:05.915208       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:53:05.915248       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:53:11.376558       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:53:11.376605       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:53:13.134433       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:53:13.134473       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:53:13.134630       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:53:13.134713       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:53:13.134761       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:53:13.134782       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:53:13.134800       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:53:13.134818       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:53:13.134837       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:53:13.134866       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:53:13.134882       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:53:13.134902       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:53:13.135071       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:53:13.135111       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:53:13.135131       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 05:53:19.895317       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:53:19.895358       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:53:22.336536       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:53:22.336589       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:53:24.470490       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:53:24.470527       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:53:28.142531       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:53:28.142559       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:53:28.142637       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:53:28.142724       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:53:28.142751       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:53:28.142771       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:53:28.142789       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:53:28.142803       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:53:28.142817       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:53:28.142949       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:53:28.143008       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:53:28.143033       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:53:28.143062       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:53:28.143109       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:53:28.143133       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 05:53:29.384187       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:53:29.384233       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:53:29.813673       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:53:29.813719       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:53:34.870989       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:53:34.871034       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:53:38.226421       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:53:38.226464       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:53:38.721667       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:53:38.721703       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:53:41.908595       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:53:41.908636       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:53:43.156108       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:53:43.156198       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:53:43.156337       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:53:43.156515       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:53:43.156559       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:53:43.156580       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:53:43.156597       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:53:43.156621       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:53:43.156650       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:53:43.156669       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:53:43.156691       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:53:43.156706       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:53:43.156840       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:53:43.156873       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:53:43.156892       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 05:53:43.684452       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:53:43.684496       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:53:46.829687       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:53:46.829738       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:53:55.882511       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:53:55.882553       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:53:58.160903       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:53:58.160968       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:53:58.160989       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:53:58.163686       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:53:58.163764       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:53:58.163800       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:53:58.163818       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:53:58.163835       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:53:58.163863       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:53:58.164059       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:53:58.164129       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:53:58.164162       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:53:58.164185       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:53:58.164200       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:53:58.164214       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 05:53:58.540153       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:53:58.540203       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:54:04.066624       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:54:04.066670       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:54:10.076291       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:54:10.076340       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:54:11.254742       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:54:11.254793       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:54:13.174208       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:54:13.174229       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:54:13.174210       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:54:13.174249       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:54:13.174291       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:54:13.174311       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:54:13.174327       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:54:13.174343       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:54:13.174361       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:54:13.174376       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:54:13.174423       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:54:13.174446       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:54:13.174460       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:54:13.174482       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:54:13.174377       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
W0508 05:54:19.020659       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:54:19.020703       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:54:22.483203       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:54:22.483246       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:54:26.143042       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:54:26.143091       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:54:28.178707       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:54:28.178806       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:54:28.178835       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:54:28.180850       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:54:28.180959       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:54:28.180996       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:54:28.181014       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:54:28.181028       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:54:28.181044       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:54:28.181549       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:54:28.181793       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:54:28.181992       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:54:28.182019       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:54:28.182034       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:54:28.182049       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 05:54:31.910916       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:54:31.910977       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:54:32.219090       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:54:32.219127       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:54:32.792559       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:54:32.792599       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:54:34.996119       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:54:34.996158       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:54:43.190008       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:54:43.190091       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:54:43.190573       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:54:43.190609       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:54:43.190627       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:54:43.190645       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:54:43.190810       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:54:43.190854       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:54:43.190906       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:54:43.190937       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:54:43.190956       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:54:43.190971       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:54:43.190993       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:54:43.191017       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:54:43.191039       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 05:54:43.775003       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:54:43.775045       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:54:52.327032       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:54:52.327084       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:54:57.335313       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:54:57.335372       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:54:58.196633       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:54:58.196727       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:54:58.196776       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:54:58.197370       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:54:58.197434       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:54:58.197460       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:54:58.197476       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:54:58.197490       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:54:58.197531       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:54:58.197695       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:54:58.197781       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:54:58.198143       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:54:58.198170       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:54:58.198185       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:54:58.198200       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 05:55:01.170998       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:55:01.171043       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:55:02.939282       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:55:02.939326       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:55:13.201057       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:55:13.201361       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:55:13.201414       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:55:13.202648       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:55:13.202741       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:55:13.203112       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:55:13.203139       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:55:13.203155       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:55:13.203172       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:55:13.203332       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:55:13.203533       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:55:13.203573       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:55:13.203619       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:55:13.203673       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:55:13.203701       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 05:55:13.482625       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:55:13.482685       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:55:15.420885       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:55:15.420926       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:55:23.323778       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:55:23.323825       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:55:25.248737       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:55:25.248780       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:55:28.206688       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:55:28.206999       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:55:28.207029       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:55:28.208026       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:55:28.208578       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:55:28.208605       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:55:28.208952       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:55:28.209035       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:55:28.209057       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:55:28.209198       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:55:28.209354       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:55:28.209421       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:55:28.209443       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:55:28.209459       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:55:28.209475       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 05:55:28.989545       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:55:28.989586       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:55:30.212940       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:55:30.212980       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:55:33.206550       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:55:33.206604       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:55:38.092031       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:55:38.092076       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:55:43.219059       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:55:43.219570       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:55:43.219594       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:55:43.219900       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:55:43.219928       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:55:43.219946       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:55:43.220495       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:55:43.220734       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:55:43.220918       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:55:43.220946       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:55:43.220961       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:55:43.220978       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:55:43.221120       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:55:43.221305       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:55:43.221327       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 05:55:46.864592       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:55:46.864637       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:55:51.212668       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:55:51.212732       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:55:58.232775       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:55:58.233193       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:55:58.233199       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:55:58.233239       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:55:58.233261       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:55:58.233283       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:55:58.233322       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:55:58.233361       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:55:58.233436       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:55:58.233622       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:55:58.233681       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:55:58.233707       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:55:58.233905       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:55:58.233947       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:55:58.233975       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 05:55:58.407929       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:55:58.407977       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:56:00.020699       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:56:00.020760       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:56:04.682859       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:56:04.682900       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:56:08.195532       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:56:08.195571       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:56:13.243506       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:56:13.243772       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:56:13.243796       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:56:13.243972       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:56:13.243974       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:56:13.244005       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:56:13.244024       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:56:13.244039       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:56:13.244054       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:56:13.244218       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:56:13.244255       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:56:13.244274       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:56:13.244353       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:56:13.244380       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:56:13.244435       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 05:56:20.873162       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:56:20.873202       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:56:22.742982       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:56:22.743025       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:56:22.853070       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:56:22.853107       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:56:23.755049       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:56:23.755091       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:56:24.414990       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:56:24.415043       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:56:28.091452       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:56:28.091489       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:56:28.248628       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:56:28.248821       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:56:28.248871       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:56:28.248895       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:56:28.248911       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:56:28.248926       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:56:28.251162       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:56:28.251173       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:56:28.251216       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:56:28.251234       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:56:28.251251       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:56:28.251273       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:56:28.251724       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:56:28.251795       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:56:28.251821       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 05:56:32.868528       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:56:32.868570       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:56:39.431495       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:56:39.431543       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:56:42.300288       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:56:42.300332       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:56:43.259841       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:56:43.259944       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:56:43.259959       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:56:43.259988       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:56:43.260007       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:56:43.260022       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:56:43.260028       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:56:43.260037       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:56:43.260055       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:56:43.260080       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:56:43.260094       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:56:43.260110       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:56:43.260370       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:56:43.260428       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:56:43.260448       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 05:56:45.637654       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:56:45.637693       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:56:46.820363       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:56:46.820439       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:56:54.291868       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:56:54.291910       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:56:55.673192       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:56:55.673234       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:56:58.264650       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:56:58.265146       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:56:58.265178       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:56:58.265311       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:56:58.265357       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:56:58.265411       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:56:58.266085       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:56:58.266091       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:56:58.266117       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:56:58.266124       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:56:58.266145       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:56:58.266160       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:56:58.266175       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:56:58.266190       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:56:58.266208       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 05:57:13.028861       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:57:13.028905       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:57:13.275522       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:57:13.276438       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:57:13.276466       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:57:13.276854       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:57:13.276939       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:57:13.277284       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:57:13.277314       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:57:13.277330       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:57:13.277346       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:57:13.277523       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:57:13.277545       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:57:13.277596       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:57:13.277631       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:57:13.277661       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:57:13.277694       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 05:57:16.373203       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:57:16.373244       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:57:16.502709       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:57:16.502749       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:57:16.573911       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:57:16.573945       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:57:25.130286       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:57:25.130338       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:57:27.290215       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:57:27.290270       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:57:28.279563       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:57:28.279611       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:57:28.279653       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:57:28.280764       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:57:28.280931       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:57:28.280959       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:57:28.283112       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:57:28.283334       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:57:28.283363       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:57:28.283844       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:57:28.284092       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:57:28.284287       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:57:28.284312       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:57:28.284327       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:57:28.284344       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 05:57:32.024142       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:57:32.024181       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:57:32.610032       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:57:32.610086       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:57:32.948069       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:57:32.948126       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:57:43.291057       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:57:43.291119       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:57:43.291140       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:57:43.291574       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:57:43.291581       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:57:43.291599       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:57:43.291628       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:57:43.291637       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:57:43.291651       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:57:43.291667       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:57:43.291681       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:57:43.291696       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:57:43.291719       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:57:43.291734       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:57:43.291748       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 05:57:46.425413       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:57:46.425455       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:57:48.269748       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:57:48.269785       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:57:53.443048       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:57:53.443088       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:57:58.295440       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:57:58.295562       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:57:58.295597       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:57:58.296819       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:57:58.296819       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:57:58.296892       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:57:58.296915       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:57:58.296938       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:57:58.296956       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:57:58.297042       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:57:58.297076       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:57:58.297093       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:57:58.297183       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:57:58.297231       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:57:58.297259       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 05:57:58.476090       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:57:58.476133       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:58:06.621774       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:58:06.621815       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:58:06.785817       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:58:06.785854       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:58:12.563964       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:58:12.564005       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:58:13.307189       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:58:13.307523       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:58:13.307546       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:58:13.307631       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:58:13.307697       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:58:13.307828       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:58:13.307876       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:58:13.307898       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:58:13.307918       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:58:13.307944       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:58:13.307963       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:58:13.307982       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:58:13.308144       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:58:13.308452       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:58:13.308485       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 05:58:13.567054       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:58:13.567093       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:58:16.306119       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:58:16.306163       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:58:18.578589       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:58:18.578631       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:58:22.634109       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:58:22.634149       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:58:23.030002       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:58:23.030074       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:58:26.113348       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:58:26.113419       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:58:28.313221       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:58:28.313737       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:58:28.313776       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:58:28.316157       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:58:28.316249       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:58:28.316249       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:58:28.316309       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:58:28.316322       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:58:28.316346       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:58:28.316408       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:58:28.316449       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:58:28.316476       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:58:28.316501       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:58:28.316537       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:58:28.316566       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 05:58:38.797592       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:58:38.797635       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:58:39.963205       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:58:39.963243       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:58:43.325901       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:58:43.325981       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:58:43.326035       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:58:43.326096       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:58:43.325907       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:58:43.326144       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:58:43.326166       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:58:43.326184       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:58:43.326200       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:58:43.326216       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:58:43.326231       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:58:43.326245       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:58:43.326260       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:58:43.326273       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:58:43.326290       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 05:58:43.489878       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:58:43.489916       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:58:46.086731       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:58:46.086773       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:58:48.714337       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:58:48.714414       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:58:50.601344       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:58:50.601439       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:58:58.329877       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:58:58.330154       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:58:58.330185       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:58:58.332811       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:58:58.333034       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:58:58.333058       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:58:58.333377       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:58:58.333641       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:58:58.333829       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:58:58.333850       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:58:58.333865       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:58:58.333883       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:58:58.334027       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:58:58.334215       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:58:58.334236       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 05:58:59.068232       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:58:59.068271       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:59:06.468725       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:59:06.468803       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:59:13.345680       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:59:13.346100       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:59:13.346132       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:59:13.347173       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:59:13.347244       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:59:13.347266       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:59:13.347577       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:59:13.347584       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:59:13.347614       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:59:13.347645       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:59:13.347680       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:59:13.347700       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:59:13.347727       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:59:13.347756       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:59:13.347653       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
W0508 05:59:15.061514       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:59:15.061551       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:59:15.126094       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:59:15.126143       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:59:16.088613       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:59:16.088657       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:59:17.049781       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:59:17.049821       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:59:21.059736       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:59:21.059784       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:59:26.831117       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:59:26.831168       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:59:28.352335       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:59:28.352476       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:59:28.353727       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:59:28.353756       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:59:28.353775       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:59:28.353791       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:59:28.353975       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:59:28.354064       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:59:28.354135       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:59:28.354676       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:59:28.354706       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:59:28.354724       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:59:28.354740       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:59:28.354754       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:59:28.354770       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 05:59:29.021237       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:59:29.021285       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:59:36.644100       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:59:36.644146       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:59:39.641198       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:59:39.641270       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:59:43.364201       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:59:43.364695       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:59:43.364736       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:59:43.364838       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:59:43.364849       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:59:43.364909       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:59:43.364910       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:59:43.364937       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:59:43.364954       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:59:43.364971       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:59:43.364987       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:59:43.365001       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:59:43.365008       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:59:43.365043       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:59:43.365079       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 05:59:52.279793       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:59:52.279842       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:59:54.634698       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:59:54.634751       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:59:55.043653       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:59:55.043693       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:59:55.114129       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:59:55.114166       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:59:55.497517       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:59:55.497574       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 05:59:55.967300       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:59:55.967354       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 05:59:58.371269       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:59:58.371689       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:59:58.371724       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:59:58.372379       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:59:58.372670       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:59:58.372700       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 05:59:58.372915       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:59:58.372963       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 05:59:58.372975       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 05:59:58.373002       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:59:58.373021       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:59:58.373040       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:59:58.373055       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:59:58.373071       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 05:59:58.373088       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:00:06.781298       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:00:06.781364       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:00:09.501219       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:00:09.501262       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:00:13.376545       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:00:13.376640       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:00:13.376882       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:00:13.376908       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:00:13.376925       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:00:13.376942       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:00:13.377579       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:00:13.377802       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:00:13.377832       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:00:13.377994       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:00:13.378217       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:00:13.378452       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:00:13.378492       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:00:13.378509       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:00:13.378527       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:00:14.291650       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:00:14.291709       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:00:21.252602       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:00:21.252646       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:00:24.551816       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:00:24.551855       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:00:28.382510       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:00:28.382510       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:00:28.382582       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:00:28.382606       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:00:28.382623       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:00:28.382639       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:00:28.383238       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:00:28.383418       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:00:28.383443       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:00:28.384076       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:00:28.384156       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:00:28.384168       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:00:28.384185       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:00:28.384202       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:00:28.384219       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:00:29.436800       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:00:29.436839       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:00:36.274910       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:00:36.274952       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:00:36.487269       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:00:36.487309       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:00:41.358562       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:00:41.358613       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:00:43.393154       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:00:43.393156       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:00:43.393220       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:00:43.393239       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:00:43.393255       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:00:43.393273       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:00:43.393505       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:00:43.393534       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:00:43.393552       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:00:43.393739       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:00:43.393785       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:00:43.393858       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:00:43.393887       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:00:43.393903       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:00:43.393921       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:00:44.262771       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:00:44.262811       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:00:47.466892       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:00:47.466933       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:00:52.262428       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:00:52.262469       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:00:53.164787       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:00:53.164832       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:00:58.398072       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:00:58.398361       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:00:58.398411       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:00:58.399733       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:00:58.399763       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:00:58.399781       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:00:58.399803       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:00:58.399818       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:00:58.399835       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:00:58.400156       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:00:58.400192       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:00:58.400214       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:00:58.400279       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:00:58.400304       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:00:58.400338       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:01:02.814739       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:01:02.814780       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:01:05.035572       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:01:05.035611       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:01:08.474983       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:01:08.475023       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:01:09.329992       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:01:09.330031       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:01:13.410518       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:01:13.410699       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:01:13.410732       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:01:13.410883       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:01:13.410908       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:01:13.410946       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:01:13.410984       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:01:13.411019       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:01:13.411048       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:01:13.411086       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:01:13.411114       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:01:13.411116       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:01:13.411157       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:01:13.411174       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:01:13.411193       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:01:14.527245       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:01:14.527286       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:01:17.752270       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:01:17.752342       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:01:18.638233       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:01:18.638273       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:01:24.071237       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:01:24.071279       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:01:28.376916       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:01:28.376957       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:01:28.415624       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:01:28.415722       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:01:28.415765       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:01:28.415785       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:01:28.415800       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:01:28.415815       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:01:28.416152       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:01:28.416195       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:01:28.416438       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:01:28.416495       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:01:28.416533       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:01:28.416568       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:01:28.416599       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:01:28.416631       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:01:28.416661       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:01:30.983696       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:01:30.983743       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:01:32.748686       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:01:32.748728       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:01:41.842660       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:01:41.842698       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:01:43.426611       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:01:43.426766       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:01:43.426793       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:01:43.426950       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:01:43.427004       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:01:43.427141       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:01:43.427260       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:01:43.427292       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:01:43.427324       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:01:43.427342       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:01:43.427364       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:01:43.427379       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:01:43.427412       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:01:43.427429       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:01:43.427445       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:01:45.288075       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:01:45.288115       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:01:52.808503       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:01:52.808548       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:01:53.931357       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:01:53.931425       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:01:55.751603       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:01:55.751639       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:01:58.431641       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:01:58.431704       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:01:58.431725       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:01:58.433100       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:01:58.433108       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:01:58.433154       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:01:58.433172       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:01:58.433188       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:01:58.433211       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:01:58.434189       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:01:58.434459       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:01:58.434476       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:01:58.434499       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:01:58.434515       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:01:58.434529       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:02:07.426903       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:02:07.426940       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:02:08.743807       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:02:08.743850       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:02:12.942343       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:02:12.942381       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:02:13.443834       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:02:13.444119       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:02:13.444144       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:02:13.444283       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:02:13.444560       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:02:13.444742       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:02:13.444763       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:02:13.444780       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:02:13.444795       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:02:13.444954       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:02:13.445222       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:02:13.445455       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:02:13.445479       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:02:13.445495       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:02:13.445510       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:02:14.273824       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:02:14.273864       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:02:15.102037       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:02:15.102078       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:02:15.728676       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:02:15.728715       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:02:28.448461       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:02:28.448753       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:02:28.448782       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:02:28.448935       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:02:28.449131       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:02:28.449160       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:02:28.449305       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:02:28.449546       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:02:28.449576       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:02:28.451170       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:02:28.451368       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:02:28.451418       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:02:28.451437       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:02:28.451455       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:02:28.451478       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:02:29.255555       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:02:29.255598       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:02:31.154476       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:02:31.154526       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:02:40.932895       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:02:40.932937       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:02:43.461213       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:02:43.461275       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:02:43.461312       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:02:43.461731       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:02:43.461816       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:02:43.461822       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:02:43.461891       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:02:43.461909       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:02:43.461925       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:02:43.462086       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:02:43.462159       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:02:43.462192       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:02:43.462216       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:02:43.462231       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:02:43.462247       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:02:43.736794       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:02:43.736839       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:02:49.277126       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:02:49.277180       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:02:51.411120       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:02:51.411169       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:02:53.466062       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:02:53.466104       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:02:54.513617       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:02:54.513673       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:02:55.252343       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:02:55.252405       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:02:58.465946       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:02:58.466248       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:02:58.466281       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:02:58.467086       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:02:58.467208       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:02:58.467229       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:02:58.467672       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:02:58.467782       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:02:58.467823       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:02:58.467841       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:02:58.467856       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:02:58.467872       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:02:58.467999       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:02:58.468110       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:02:58.468138       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:03:00.210988       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:03:00.211027       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:03:10.330712       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:03:10.330773       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:03:13.478470       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:03:13.478740       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:03:13.478764       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:03:13.478939       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:03:13.479161       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:03:13.479402       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:03:13.479426       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:03:13.479442       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:03:13.479460       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:03:13.479653       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:03:13.479891       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:03:13.480078       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:03:13.480099       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:03:13.480116       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:03:13.480132       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:03:14.567876       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:03:14.567922       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:03:22.077695       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:03:22.077735       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:03:23.376473       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:03:23.376521       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:03:24.182203       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:03:24.182247       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:03:25.863735       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:03:25.863789       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:03:27.063549       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:03:27.063589       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:03:28.483027       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:03:28.483531       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:03:28.483567       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:03:28.483695       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:03:28.483724       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:03:28.483742       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:03:28.484776       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:03:28.484839       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:03:28.484862       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:03:28.485004       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:03:28.485031       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:03:28.485049       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:03:28.485433       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:03:28.485471       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:03:28.485492       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:03:31.085229       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:03:31.085304       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:03:34.677010       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:03:34.677049       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:03:43.494243       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:03:43.494320       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:03:43.494367       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:03:43.494409       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:03:43.494428       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:03:43.494446       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:03:43.495080       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:03:43.495089       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:03:43.495121       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:03:43.495140       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:03:43.495156       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:03:43.495175       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:03:43.495374       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:03:43.495420       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:03:43.495437       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:03:48.785157       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:03:48.785201       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:03:49.211727       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:03:49.211766       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:03:52.017858       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:03:52.017894       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:03:57.208629       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:03:57.208669       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:03:58.501225       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:03:58.501225       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:03:58.501496       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:03:58.501577       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:03:58.501619       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:03:58.501628       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:03:58.501636       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:03:58.501655       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:03:58.501670       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:03:58.501684       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:03:58.501699       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:03:58.501714       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:03:58.501731       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:03:58.501746       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:03:58.501762       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:04:01.362670       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:04:01.362727       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:04:02.511288       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:04:02.511330       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:04:04.321696       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:04:04.321736       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:04:13.512403       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:04:13.512500       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:04:13.512564       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:04:13.513044       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:04:13.513066       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:04:13.513082       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:04:13.513098       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:04:13.513112       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:04:13.513131       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:04:13.513285       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:04:13.513312       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:04:13.513331       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:04:13.513428       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:04:13.513463       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:04:13.513480       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:04:14.341770       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:04:14.341812       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:04:19.739826       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:04:19.739872       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:04:21.927367       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:04:21.927452       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:04:23.242461       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:04:23.242524       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:04:23.399848       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:04:23.399889       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:04:28.517465       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:04:28.517529       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:04:28.517551       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:04:28.518476       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:04:28.518661       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:04:28.518684       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:04:28.520082       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:04:28.520104       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:04:28.520090       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:04:28.520132       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:04:28.520149       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:04:28.520163       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:04:28.520179       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:04:28.520193       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:04:28.520208       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:04:33.599532       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:04:33.599574       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:04:43.529910       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:04:43.530185       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:04:43.530208       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:04:43.530452       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:04:43.530493       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:04:43.530514       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:04:43.530583       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:04:43.530617       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:04:43.530636       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:04:43.530651       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:04:43.530659       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:04:43.530713       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:04:43.530815       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:04:43.530840       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:04:43.530864       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:04:43.889794       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:04:43.889850       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:04:50.213674       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:04:50.213726       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:04:55.187125       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:04:55.187167       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:04:57.377357       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:04:57.377426       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:04:58.535282       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:04:58.535730       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:04:58.535777       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:04:58.537165       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:04:58.537297       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:04:58.537419       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:04:58.538078       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:04:58.538112       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:04:58.538129       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:04:58.538145       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:04:58.538159       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:04:58.538178       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:04:58.538347       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:04:58.538458       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:04:58.538492       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:05:02.169954       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:05:02.169992       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:05:02.202374       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:05:02.202423       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:05:06.352078       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:05:06.352119       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:05:07.820025       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:05:07.820065       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:05:13.067154       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:05:13.067194       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:05:13.539995       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:05:13.540287       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:05:13.540319       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:05:13.542762       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:05:13.542833       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:05:13.542857       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:05:13.543061       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:05:13.543155       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:05:13.543296       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:05:13.543328       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:05:13.543348       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:05:13.543363       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:05:13.543380       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:05:13.543416       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:05:13.543434       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:05:20.248027       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:05:20.248069       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:05:23.269343       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:05:23.269407       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:05:25.157797       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:05:25.157842       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:05:28.544693       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:05:28.545606       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:05:28.545635       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:05:28.547362       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:05:28.547611       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:05:28.547635       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:05:28.548022       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:05:28.548222       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:05:28.548244       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:05:28.548425       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:05:28.548630       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:05:28.548652       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:05:28.548818       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:05:28.549003       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:05:28.549023       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:05:31.167701       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:05:31.167748       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:05:38.679124       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:05:38.679165       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:05:43.556528       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:05:43.556604       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:05:43.556628       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:05:43.557797       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:05:43.557799       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:05:43.557887       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:05:43.557917       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:05:43.557934       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:05:43.557952       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:05:43.557970       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:05:43.557985       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:05:43.558000       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:05:43.558560       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:05:43.558609       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:05:43.558630       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:05:45.585670       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:05:45.585715       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:05:52.199591       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:05:52.199639       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:05:55.552560       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:05:55.552603       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:05:56.832675       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:05:56.832724       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:05:56.905088       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:05:56.905117       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:05:57.300344       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:05:57.300422       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:05:58.561573       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:05:58.561848       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:05:58.561872       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:05:58.562304       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:05:58.562539       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:05:58.562566       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:05:58.563572       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:05:58.563786       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:05:58.563809       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:05:58.563824       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:05:58.563841       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:05:58.563789       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:05:58.563921       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:05:58.563946       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:05:58.563963       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:06:10.919543       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:06:10.919585       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:06:13.573247       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:06:13.573551       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:06:13.573586       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:06:13.573652       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:06:13.573750       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:06:13.573773       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:06:13.574195       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:06:13.574549       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:06:13.574725       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:06:13.574761       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:06:13.574781       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:06:13.574797       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:06:13.574815       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:06:13.574830       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:06:13.574845       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:06:13.821957       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:06:13.822005       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:06:19.715727       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:06:19.715765       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:06:20.071979       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:06:20.072019       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:06:21.740315       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:06:21.740355       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:06:23.468679       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:06:23.468736       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:06:27.755021       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:06:27.755060       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:06:28.578865       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:06:28.579144       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:06:28.579176       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:06:28.579493       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:06:28.579774       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:06:28.579982       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:06:28.580010       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:06:28.580028       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:06:28.580057       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:06:28.581083       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:06:28.581207       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:06:28.581526       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:06:28.581570       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:06:28.581604       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:06:28.581630       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:06:30.640024       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:06:30.640068       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:06:34.501150       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:06:34.501197       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:06:38.663691       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:06:38.663734       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:06:43.590689       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:06:43.590769       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:06:43.590848       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:06:43.590901       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:06:43.590960       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:06:43.591830       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:06:43.591858       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:06:43.591876       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:06:43.591891       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:06:43.591903       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:06:43.591918       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:06:43.591934       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:06:43.591949       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:06:43.591962       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:06:43.591978       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:06:48.705127       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:06:48.705166       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:06:54.479307       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:06:54.479346       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:06:55.754124       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:06:55.754167       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:06:55.765177       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:06:55.765200       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:06:58.595131       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:06:58.595296       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:06:58.595319       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:06:58.596576       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:06:58.596588       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:06:58.596601       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:06:58.596626       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:06:58.596644       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:06:58.596654       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:06:58.596659       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:06:58.596680       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:06:58.596697       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:06:58.596720       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:06:58.596735       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:06:58.596750       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:06:59.901623       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:06:59.901673       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:07:04.029322       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:07:04.029363       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:07:13.607022       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:07:13.607114       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:07:13.607174       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:07:13.607760       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:07:13.607784       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:07:13.607802       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:07:13.607818       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:07:13.607834       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:07:13.607849       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:07:13.607900       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:07:13.607926       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:07:13.608052       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:07:13.608074       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:07:13.608089       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:07:13.608103       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:07:13.918057       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:07:13.918098       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:07:17.416315       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:07:17.416373       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:07:27.678120       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:07:27.678169       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:07:27.897560       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:07:27.897602       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:07:28.285195       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:07:28.285265       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:07:28.611183       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:07:28.612349       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:07:28.612413       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:07:28.612977       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:07:28.613106       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:07:28.613723       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:07:28.613764       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:07:28.613792       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:07:28.613819       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:07:28.614249       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:07:28.614635       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:07:28.614677       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:07:28.615076       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:07:28.615134       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:07:28.615175       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:07:30.097212       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:07:30.097271       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:07:30.889630       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:07:30.889673       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:07:37.605008       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:07:37.605048       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:07:38.909496       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:07:38.909539       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:07:43.585904       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:07:43.585967       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:07:43.624176       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:07:43.624491       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:07:43.624516       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:07:43.624693       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:07:43.624709       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:07:43.624792       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:07:43.624873       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:07:43.624909       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:07:43.624926       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:07:43.624942       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:07:43.624957       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:07:43.624971       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:07:43.624986       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:07:43.625000       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:07:43.625016       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:07:48.924351       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:07:48.924413       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:07:58.629192       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:07:58.629562       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:07:58.629590       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:07:58.629687       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:07:58.629902       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:07:58.629922       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:07:58.629928       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:07:58.629951       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:07:58.629967       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:07:58.630229       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:07:58.630260       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:07:58.630279       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:07:58.630419       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:07:58.630455       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:07:58.630476       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:08:00.906614       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:08:00.906667       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:08:07.193530       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:08:07.193574       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:08:08.786792       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:08:08.786844       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:08:09.440579       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:08:09.440629       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:08:13.642890       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:08:13.642951       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:08:13.643007       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:08:13.643029       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:08:13.643044       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:08:13.643060       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:08:13.643942       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:08:13.644000       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:08:13.644009       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:08:13.644021       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:08:13.644036       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:08:13.644074       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:08:13.644099       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:08:13.644116       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:08:13.644131       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:08:15.090345       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:08:15.090406       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:08:20.576372       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:08:20.576444       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:08:22.715596       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:08:22.715637       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:08:26.982001       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:08:26.982043       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:08:28.448607       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:08:28.448649       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:08:28.647827       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:08:28.647885       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:08:28.647827       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:08:28.647946       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:08:28.647967       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:08:28.647986       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:08:28.648001       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:08:28.648021       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:08:28.648037       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:08:28.649501       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:08:28.649508       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:08:28.649538       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:08:28.649558       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:08:28.649594       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:08:28.649614       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:08:28.904878       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:08:28.904916       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:08:29.174855       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:08:29.174892       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:08:35.185441       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:08:35.185481       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:08:43.658999       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:08:43.659274       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:08:43.659301       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:08:43.659506       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:08:43.659529       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:08:43.659723       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:08:43.659743       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:08:43.659760       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:08:43.659776       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:08:43.660836       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:08:43.660854       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:08:43.660887       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:08:43.660905       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:08:43.660920       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:08:43.660937       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:08:45.266285       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:08:45.266325       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:08:57.313227       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:08:57.313267       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:08:58.490853       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:08:58.490890       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:08:58.664005       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:08:58.664306       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:08:58.664336       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:08:58.664438       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:08:58.664613       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:08:58.664647       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:08:58.664667       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:08:58.664682       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:08:58.664698       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:08:58.665470       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:08:58.665506       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:08:58.665527       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:08:58.666540       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:08:58.666581       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:08:58.666601       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:08:59.226884       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:08:59.226924       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:09:06.992179       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:09:06.992231       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:09:09.238276       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:09:09.238320       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:09:11.120262       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:09:11.120299       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:09:11.828520       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:09:11.828560       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:09:13.576497       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:09:13.576538       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:09:13.675629       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:09:13.675661       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:09:13.675689       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:09:13.676369       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:09:13.676518       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:09:13.676548       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:09:13.676794       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:09:13.676852       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:09:13.676875       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:09:13.676888       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:09:13.676897       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:09:13.676926       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:09:13.676943       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:09:13.676961       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:09:13.676976       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:09:17.096535       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:09:17.096576       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:09:21.911884       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:09:21.911924       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:09:28.680651       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:09:28.680851       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:09:28.680905       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:09:28.680928       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:09:28.680945       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:09:28.680961       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:09:28.681050       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:09:28.681074       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:09:28.681090       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:09:28.681638       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:09:28.681746       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:09:28.681774       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:09:28.681788       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:09:28.681817       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:09:28.681834       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:09:40.646714       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:09:40.646754       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:09:42.461821       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:09:42.461860       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:09:43.691974       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:09:43.692048       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:09:43.692159       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:09:43.692207       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:09:43.692228       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:09:43.692244       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:09:43.693101       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:09:43.693369       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:09:43.693417       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:09:43.693502       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:09:43.693512       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:09:43.693555       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:09:43.693591       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:09:43.693609       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:09:43.693625       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:09:46.501034       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:09:46.501076       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:09:47.553824       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:09:47.553867       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:09:47.897638       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:09:47.897678       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:09:48.728517       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:09:48.728556       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:09:58.687929       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:09:58.687968       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:09:58.698647       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:09:58.698714       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:09:58.698841       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:09:58.698898       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:09:58.698931       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:09:58.698960       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:09:58.698988       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:09:58.699007       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:09:58.699024       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:09:58.699457       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:09:58.699506       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:09:58.699538       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:09:58.699926       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:09:58.700005       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:09:58.700032       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:10:06.876810       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:10:06.876849       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:10:09.008617       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:10:09.008654       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:10:09.468974       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:10:09.469014       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:10:13.704645       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:10:13.704700       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:10:13.704644       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:10:13.704804       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:10:13.704838       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:10:13.704871       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:10:13.704882       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:10:13.704901       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:10:13.704926       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:10:13.704964       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:10:13.704992       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:10:13.705020       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:10:13.705615       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:10:13.705654       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:10:13.705672       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:10:18.279247       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:10:18.279288       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:10:19.646279       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:10:19.646319       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:10:21.008948       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:10:21.008987       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:10:25.819920       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:10:25.819963       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:10:28.709467       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:10:28.709563       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:10:28.709676       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:10:28.709699       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:10:28.709714       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:10:28.709730       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:10:28.709868       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:10:28.709930       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:10:28.709985       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:10:28.710021       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:10:28.710038       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:10:28.710056       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:10:28.710106       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:10:28.710139       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:10:28.710163       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:10:31.587091       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:10:31.587132       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:10:32.300179       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:10:32.300235       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:10:36.977085       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:10:36.977126       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:10:37.032715       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:10:37.032742       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:10:43.721547       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:10:43.721634       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:10:43.721645       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:10:43.721692       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:10:43.721562       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:10:43.721734       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:10:43.721752       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:10:43.721768       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:10:43.721784       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:10:43.721808       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:10:43.721822       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:10:43.721838       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:10:43.721855       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:10:43.721869       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:10:43.721885       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:10:46.935252       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:10:46.935296       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:10:49.044757       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:10:49.044794       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:10:53.355523       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:10:53.355576       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:10:58.489154       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:10:58.489210       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:10:58.726908       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:10:58.727192       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:10:58.727216       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:10:58.728521       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:10:58.728595       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:10:58.728954       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:10:58.728983       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:10:58.728999       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:10:58.729014       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:10:58.729225       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:10:58.729458       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:10:58.729491       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:10:58.729512       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:10:58.729597       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:10:58.729620       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:11:00.890801       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:11:00.890844       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:11:06.075849       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:11:06.075886       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:11:06.825208       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:11:06.825262       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:11:12.460793       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:11:12.460834       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:11:13.738591       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:11:13.738687       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:11:13.738755       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:11:13.739295       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:11:13.739323       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:11:13.739341       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:11:13.739356       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:11:13.739373       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:11:13.739405       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:11:13.739557       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:11:13.739812       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:11:13.739851       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:11:13.739873       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:11:13.740058       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:11:13.740083       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:11:22.861796       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:11:22.861833       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:11:23.887595       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:11:23.887645       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:11:25.870499       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:11:25.870545       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:11:28.746570       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:11:28.746609       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:11:28.746648       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:11:28.746671       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:11:28.746690       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:11:28.746709       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:11:28.747717       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:11:28.747774       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:11:28.747792       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:11:28.747823       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:11:28.747843       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:11:28.747861       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:11:28.747879       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:11:28.747895       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:11:28.747912       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:11:35.454337       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:11:35.454379       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:11:37.644160       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:11:37.644199       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:11:38.030068       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:11:38.030109       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:11:39.797120       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:11:39.797175       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:11:41.299232       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:11:41.299277       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:11:43.757458       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:11:43.757537       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:11:43.757932       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:11:43.757964       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:11:43.757980       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:11:43.757996       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:11:43.758189       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:11:43.758488       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:11:43.758576       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:11:43.758612       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:11:43.758630       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:11:43.758965       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:11:43.758991       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:11:43.759006       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:11:43.759020       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:11:46.227134       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:11:46.227188       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:11:55.026721       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:11:55.026760       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:11:58.762183       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:11:58.762245       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:11:58.762267       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:11:58.763079       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:11:58.763154       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:11:58.763184       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:11:58.763200       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:11:58.763216       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:11:58.763158       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:11:58.764457       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:11:58.764677       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:11:58.764701       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:11:58.764946       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:11:58.765017       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:11:58.765043       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:12:01.318738       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:12:01.318776       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:12:03.660779       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:12:03.660823       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:12:10.982248       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:12:10.982288       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:12:11.494077       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:12:11.494129       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:12:13.045870       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:12:13.045908       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:12:13.774172       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:12:13.774250       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:12:13.774307       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:12:13.774326       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:12:13.774341       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:12:13.774357       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:12:13.774885       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:12:13.775130       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:12:13.775416       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:12:13.775604       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:12:13.775630       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:12:13.775645       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:12:13.775660       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:12:13.775674       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:12:13.775691       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:12:15.800335       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:12:15.800374       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:12:18.293870       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:12:18.293908       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:12:22.584322       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:12:22.584359       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:12:24.895706       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:12:24.895742       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:12:28.779024       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:12:28.779111       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:12:28.779547       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:12:28.779571       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:12:28.779587       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:12:28.779606       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:12:28.779743       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:12:28.779944       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:12:28.779980       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:12:28.781040       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:12:28.781125       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:12:28.781169       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:12:28.781257       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:12:28.781293       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:12:28.781311       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:12:31.582187       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:12:31.582245       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:12:33.305329       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:12:33.305367       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:12:39.470908       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:12:39.470949       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:12:39.744941       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:12:39.744981       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:12:43.790915       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:12:43.790982       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:12:43.791004       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:12:43.791262       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:12:43.791331       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:12:43.791514       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:12:43.791639       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:12:43.791676       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:12:43.791695       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:12:43.791710       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:12:43.791726       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:12:43.791740       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:12:43.791756       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:12:43.791770       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:12:43.791785       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:12:47.582107       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:12:47.582158       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:12:49.669884       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:12:49.669934       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:12:51.629934       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:12:51.629971       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:12:52.625279       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:12:52.625316       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:12:54.204056       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:12:54.204096       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:12:58.795612       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:12:58.795889       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:12:58.795913       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:12:58.797034       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:12:58.797266       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:12:58.797290       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:12:58.797439       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:12:58.797674       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:12:58.797863       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:12:58.797889       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:12:58.797907       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:12:58.797924       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:12:58.798060       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:12:58.798279       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:12:58.798306       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:13:06.331581       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:13:06.331623       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:13:13.513238       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:13:13.513292       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:13:13.809413       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:13:13.809701       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:13:13.809731       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:13:13.809898       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:13:13.810118       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:13:13.810157       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:13:13.810189       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:13:13.810274       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:13:13.810296       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:13:13.810345       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:13:13.810350       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:13:13.810381       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:13:13.810425       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:13:13.810455       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:13:13.810473       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:13:18.323595       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:13:18.323635       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:13:22.028155       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:13:22.028195       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:13:22.872453       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:13:22.872494       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:13:27.339342       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:13:27.339407       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:13:28.814017       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:13:28.814346       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:13:28.814368       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:13:28.814437       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:13:28.814654       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:13:28.814682       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:13:28.816211       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:13:28.816460       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:13:28.816482       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:13:28.816650       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:13:28.816671       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:13:28.816702       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:13:28.816721       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:13:28.816735       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:13:28.816751       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:13:29.848542       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:13:29.848585       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:13:30.095599       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:13:30.095653       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:13:30.298947       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:13:30.298984       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:13:38.840994       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:13:38.841041       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:13:42.478987       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:13:42.479030       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:13:43.826256       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:13:43.826330       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:13:43.826344       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:13:43.826346       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:13:43.826379       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:13:43.826425       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:13:43.826442       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:13:43.826458       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:13:43.826473       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:13:43.826488       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:13:43.826497       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:13:43.826509       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:13:43.826525       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:13:43.826540       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:13:43.826557       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:13:48.430525       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:13:48.430566       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:13:53.074640       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:13:53.074680       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:13:58.830678       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:13:58.830955       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:13:58.830981       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:13:58.832043       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:13:58.832088       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:13:58.832115       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:13:58.832547       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:13:58.832583       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:13:58.832601       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:13:58.832886       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:13:58.832949       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:13:58.832968       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:13:58.833246       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:13:58.833284       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:13:58.833305       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:13:59.604055       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:13:59.604097       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:14:12.140269       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:14:12.140308       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:14:12.692341       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:14:12.692429       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:14:13.845425       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:14:13.845481       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:14:13.845516       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:14:13.846105       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:14:13.846422       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:14:13.846464       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:14:13.846659       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:14:13.846991       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:14:13.847065       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:14:13.847092       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:14:13.847113       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:14:13.847144       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:14:13.847163       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:14:13.847178       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:14:13.847192       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:14:18.563012       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:14:18.563054       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:14:19.510517       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:14:19.510554       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:14:20.577824       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:14:20.577864       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:14:23.961324       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:14:23.961379       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:14:24.431173       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:14:24.431212       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:14:28.850024       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:14:28.850146       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:14:28.850181       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:14:28.851148       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:14:28.851189       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:14:28.851436       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:14:28.851461       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:14:28.851477       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:14:28.851492       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:14:28.851980       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:14:28.852221       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:14:28.852244       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:14:28.852946       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:14:28.853042       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:14:28.853065       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:14:32.782193       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:14:32.782233       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:14:32.958899       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:14:32.958945       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:14:37.817210       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:14:37.817246       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:14:39.787162       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:14:39.787200       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:14:43.861582       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:14:43.861696       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:14:43.861884       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:14:43.861916       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:14:43.861936       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:14:43.861953       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:14:43.862044       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:14:43.862045       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:14:43.862067       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:14:43.862091       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:14:43.862108       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:14:43.862146       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:14:43.862328       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:14:43.862359       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:14:43.862376       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:14:58.865815       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:14:58.865995       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:14:58.866020       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:14:58.867301       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:14:58.867318       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:14:58.867403       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:14:58.867440       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:14:58.867457       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:14:58.867473       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:14:58.867492       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:14:58.867507       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:14:58.867523       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:14:58.867809       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:14:58.867916       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:14:58.867943       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:14:59.210970       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:14:59.211010       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:15:02.022409       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:15:02.022452       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:15:02.949557       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:15:02.949599       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:15:09.301543       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:15:09.301584       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:15:10.112141       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:15:10.112183       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:15:13.871931       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:15:13.872047       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:15:13.872362       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:15:13.872414       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:15:13.872434       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:15:13.872450       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:15:13.872492       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:15:13.872537       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:15:13.872569       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:15:13.873192       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:15:13.873192       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:15:13.873252       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:15:13.873286       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:15:13.873315       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:15:13.873350       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:15:15.078570       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:15:15.078613       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:15:15.225133       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:15:15.225167       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:15:18.022212       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:15:18.022250       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:15:18.137180       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:15:18.137218       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:15:21.399556       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:15:21.399600       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:15:22.125355       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:15:22.125407       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:15:28.877777       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:15:28.877879       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:15:28.877980       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:15:28.878029       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:15:28.878064       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:15:28.878100       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:15:28.878110       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:15:28.878162       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:15:28.878201       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:15:28.878306       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:15:28.878350       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:15:28.878384       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:15:28.879109       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:15:28.879317       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:15:28.879343       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:15:29.443427       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:15:29.443471       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:15:35.510712       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:15:35.510762       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:15:40.822751       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:15:40.822801       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:15:43.888512       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:15:43.888819       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:15:43.888850       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:15:43.890922       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:15:43.891004       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:15:43.891068       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:15:43.891614       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:15:43.891644       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:15:43.891659       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:15:43.891677       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:15:43.891692       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:15:43.891707       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:15:43.891843       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:15:43.892047       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:15:43.892073       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:15:47.369898       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:15:47.369941       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:15:49.018986       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:15:49.019023       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:15:55.475262       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:15:55.475308       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:15:57.476427       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:15:57.476468       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:15:58.893721       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:15:58.893792       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:15:58.893821       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:15:58.895701       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:15:58.895747       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:15:58.895767       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:15:58.895839       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:15:58.895870       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:15:58.895888       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:15:58.896175       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:15:58.896185       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:15:58.896214       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:15:58.896230       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:15:58.896246       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:15:58.896261       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:16:03.192967       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:16:03.193014       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:16:05.932957       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:16:05.933001       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:16:11.144570       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:16:11.144618       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:16:13.269179       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:16:13.269222       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:16:13.911565       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:16:13.911563       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:16:13.911740       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:16:13.911805       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:16:13.911935       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:16:13.911987       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:16:13.912008       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:16:13.912025       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:16:13.912042       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:16:13.912092       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:16:13.912110       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:16:13.912140       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:16:13.912157       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:16:13.912172       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:16:13.912189       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:16:14.411168       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:16:14.411220       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:16:24.561671       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:16:24.561724       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:16:28.919314       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:16:28.919659       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:16:28.919684       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:16:28.920348       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:16:28.920457       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:16:28.920488       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:16:28.920508       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:16:28.920524       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:16:28.920542       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:16:28.920882       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:16:28.921056       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:16:28.921114       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:16:28.921191       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:16:28.921212       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:16:28.921231       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:16:32.453199       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:16:32.453253       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:16:36.698779       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:16:36.698825       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:16:41.739515       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:16:41.739579       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:16:43.659813       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:16:43.659881       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:16:43.933091       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:16:43.933290       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:16:43.933380       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:16:43.933427       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:16:43.933445       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:16:43.933465       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:16:43.933596       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:16:43.933624       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:16:43.933642       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:16:43.933768       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:16:43.933862       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:16:43.933900       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:16:43.933919       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:16:43.933935       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:16:43.933953       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:16:44.751837       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:16:44.751883       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:16:44.877151       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:16:44.877185       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:16:45.539410       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:16:45.539454       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:16:49.309539       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:16:49.309591       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:16:53.276188       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:16:53.276230       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:16:55.437098       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:16:55.437150       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:16:58.938009       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:16:58.938175       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:16:58.938208       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:16:58.939573       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:16:58.939628       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:16:58.939650       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:16:58.940472       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:16:58.940487       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:16:58.940623       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:16:58.940655       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:16:58.940672       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:16:58.940689       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:16:58.940704       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:16:58.940720       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:16:58.940736       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:17:13.950041       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:17:13.950238       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:17:13.950268       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:17:13.950503       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:17:13.950746       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:17:13.950959       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:17:13.950991       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:17:13.951008       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:17:13.951024       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:17:13.951208       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:17:13.951495       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:17:13.951538       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:17:13.951558       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:17:13.951576       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:17:13.951591       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:17:16.249206       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:17:16.249263       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:17:21.237010       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:17:21.237051       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:17:22.391638       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:17:22.391676       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:17:23.571774       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:17:23.571815       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:17:23.573009       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:17:23.573050       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:17:28.956354       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:17:28.956437       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:17:28.956534       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:17:28.956619       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:17:28.956647       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:17:28.956665       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:17:28.956682       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:17:28.956697       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:17:28.956713       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:17:28.957284       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:17:28.957335       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:17:28.957355       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:17:28.957441       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:17:28.957460       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:17:28.957490       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:17:31.055642       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:17:31.055688       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:17:37.807538       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:17:37.807585       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:17:40.339920       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:17:40.339962       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:17:40.788421       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:17:40.788472       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:17:40.960374       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:17:40.960436       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:17:43.967450       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:17:43.967572       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:17:43.967598       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:17:43.968183       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:17:43.968219       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:17:43.968238       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:17:43.968256       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:17:43.968289       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:17:43.968309       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:17:43.969053       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:17:43.969053       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:17:43.969078       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:17:43.969099       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:17:43.969128       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:17:43.969148       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:17:44.035906       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:17:44.035943       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:17:58.973023       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:17:58.973513       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:17:58.973558       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:17:58.973698       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:17:58.973830       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:17:58.973875       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:17:58.973942       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:17:58.973973       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:17:58.973974       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:17:58.974018       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:17:58.974049       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:17:58.974077       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:17:58.974266       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:17:58.974302       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:17:58.974322       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:18:06.001342       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:18:06.001406       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:18:09.833881       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:18:09.833929       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:18:10.569269       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:18:10.569330       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:18:13.985902       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:18:13.986217       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:18:13.986246       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:18:13.986505       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:18:13.986846       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:18:13.986909       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:18:13.986934       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:18:13.986953       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:18:13.986969       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:18:13.986987       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:18:13.987004       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:18:13.987021       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:18:13.987156       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:18:13.987189       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:18:13.987206       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:18:14.551806       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:18:14.551856       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:18:17.907808       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:18:17.907868       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:18:19.110239       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:18:19.110307       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:18:19.720961       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:18:19.721012       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:18:28.990658       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:18:28.990785       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:18:28.990826       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:18:28.991943       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:18:28.992182       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:18:28.992209       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:18:28.993155       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:18:28.993204       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:18:28.993236       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:18:28.993255       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:18:28.993270       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:18:28.993286       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:18:28.993495       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:18:28.993531       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:18:28.993552       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:18:29.979468       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:18:29.979519       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:18:34.097162       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:18:34.097208       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:18:35.557822       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:18:35.557869       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:18:37.843040       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:18:37.843085       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:18:44.002165       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:18:44.002249       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:18:44.002700       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:18:44.002729       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:18:44.002749       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:18:44.002766       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:18:44.002847       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:18:44.002882       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:18:44.002887       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:18:44.002919       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:18:44.002939       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:18:44.002955       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:18:44.002971       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:18:44.002987       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:18:44.003004       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:18:50.650634       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:18:50.650689       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:18:56.669455       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:18:56.669499       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:18:57.949819       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:18:57.949875       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:18:59.008136       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:18:59.008447       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:18:59.008472       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:18:59.008626       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:18:59.008837       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:18:59.008862       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:18:59.009956       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:18:59.010020       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:18:59.010043       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:18:59.010554       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:18:59.010586       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:18:59.010609       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:18:59.010684       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:18:59.010711       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:18:59.010743       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:19:02.344284       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:19:02.344329       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:19:02.427996       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:19:02.428046       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:19:05.621986       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:19:05.622055       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:19:08.700135       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:19:08.700181       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:19:10.386986       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:19:10.387029       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:19:14.021456       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:19:14.021554       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:19:14.021626       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:19:14.021721       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:19:14.021791       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:19:14.022152       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:14.022179       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:14.022199       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:14.022219       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:14.022251       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:14.022271       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:14.022288       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:14.022305       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:14.022321       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:14.022341       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:19:21.349860       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:19:21.349928       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:19:25.703411       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:19:25.703459       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:19:27.214945       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:19:27.215002       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:19:29.026166       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:19:29.026492       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:29.026516       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:19:29.028428       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:19:29.028497       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:19:29.028783       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:19:29.028858       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:19:29.028889       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:29.028907       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:29.028924       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:29.028940       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:29.028954       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:29.028974       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:29.028988       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:29.029003       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:19:30.552286       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:19:30.552327       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:19:32.199269       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:19:32.199313       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:19:39.760739       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:19:39.760780       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:19:44.038153       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:19:44.038476       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:44.038527       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:19:44.038695       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:19:44.038988       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:19:44.039261       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:19:44.039550       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:19:44.039753       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:44.039781       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:44.039797       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:44.039814       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:44.039829       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:44.039844       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:44.039859       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:44.039876       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:19:46.436234       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:19:46.436283       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:19:51.030459       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:19:51.030503       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:19:51.127434       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:19:51.127481       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:19:55.472628       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:19:55.472672       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:19:59.044489       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:19:59.044628       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:19:59.044858       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:19:59.045254       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:59.045295       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:59.045321       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:59.045347       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:59.045369       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:59.045414       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:19:59.045933       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:19:59.046013       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:59.046037       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:19:59.046407       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:19:59.046492       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:19:59.046524       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:19:59.284803       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:19:59.284851       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:20:01.768046       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:20:01.768090       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:20:02.131177       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:20:02.131223       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:20:06.998262       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:20:06.998308       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:20:12.045293       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:20:12.045331       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:20:14.049190       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:20:14.049492       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:20:14.049521       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:20:14.050421       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:20:14.050445       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:20:14.050464       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:20:14.050482       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:20:14.050498       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:20:14.050503       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:20:14.050515       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:20:14.050530       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:20:14.050546       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:20:14.050828       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:20:14.050859       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:20:14.050875       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:20:22.188480       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:20:22.188522       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:20:26.212161       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:20:26.212208       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:20:29.054781       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:20:29.054900       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:20:29.055159       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:20:29.055204       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:20:29.055223       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:20:29.055239       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:20:29.055256       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:20:29.055272       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:20:29.055287       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:20:29.056143       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:20:29.056373       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:20:29.056423       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:20:29.056566       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:20:29.056771       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:20:29.056801       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:20:30.119414       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:20:30.119480       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:20:32.729684       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:20:32.729724       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:20:36.428836       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:20:36.428877       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:20:39.366620       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:20:39.366658       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:20:44.067069       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:20:44.067172       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:20:44.067233       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:20:44.067356       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:20:44.067478       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:20:44.067510       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:20:44.067517       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:20:44.067531       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:20:44.067549       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:20:44.067570       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:20:44.067585       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:20:44.067601       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:20:44.067616       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:20:44.067630       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:20:44.067645       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:20:46.555942       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:20:46.555983       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:20:57.910224       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:20:57.910266       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:20:58.172482       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:20:58.172525       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:20:59.072441       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:20:59.072658       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:20:59.072682       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:20:59.073205       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:20:59.073315       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:20:59.073345       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:20:59.073508       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:20:59.073623       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:20:59.073753       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:20:59.073782       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:20:59.073801       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:20:59.073819       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:20:59.073835       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:20:59.073849       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:20:59.073863       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:21:01.804404       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:21:01.804448       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:21:07.924776       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:21:07.924817       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:21:08.557574       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:21:08.557617       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:21:11.712540       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:21:11.712580       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:21:14.083779       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:21:14.083838       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:21:14.083864       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:21:14.084148       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:21:14.084281       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:21:14.084493       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:21:14.084549       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:21:14.084578       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:21:14.084596       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:21:14.084611       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:21:14.084633       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:21:14.084649       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:21:14.084665       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:21:14.084689       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:21:14.084704       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:21:16.270293       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:21:16.270331       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:21:16.388635       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:21:16.388667       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:21:20.248039       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:21:20.248080       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:21:25.426655       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:21:25.426704       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:21:29.089033       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:21:29.089350       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:21:29.089379       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:21:29.090055       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:21:29.090456       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:21:29.090502       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:21:29.090809       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:21:29.090847       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:21:29.090866       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:21:29.091946       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:21:29.091978       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:21:29.092017       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:21:29.092044       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:21:29.092079       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:21:29.092118       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:21:32.677646       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:21:32.677688       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:21:40.116838       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:21:40.116881       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:21:41.113538       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:21:41.113579       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:21:41.212310       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:21:41.212338       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:21:44.101021       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:21:44.101329       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:21:44.101354       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:21:44.101849       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:21:44.101868       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:21:44.101888       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:21:44.101907       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:21:44.101925       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:21:44.101942       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:21:44.101946       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:21:44.101964       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:21:44.101980       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:21:44.102002       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:21:44.102017       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:21:44.101985       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
W0508 06:21:49.477849       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:21:49.477889       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:21:51.690648       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:21:51.690687       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:21:51.761299       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:21:51.761325       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:21:55.444289       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:21:55.444327       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:21:59.105562       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:21:59.105855       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:21:59.105887       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:21:59.106978       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:21:59.107228       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:21:59.107255       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:21:59.107331       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:21:59.107436       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:21:59.107519       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:21:59.107552       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:21:59.107572       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:21:59.107592       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:21:59.107608       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:21:59.107622       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:21:59.107641       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:21:59.774531       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:21:59.774572       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:22:01.630177       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:22:01.630213       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:22:05.291283       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:22:05.291327       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:22:14.117808       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:22:14.117885       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:22:14.117891       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:22:14.117808       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:22:14.117944       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:22:14.117964       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:22:14.117980       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:22:14.117998       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:22:14.118022       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:22:14.118036       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:22:14.118046       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:22:14.118052       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:22:14.118081       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:22:14.118101       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:22:14.118138       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:22:15.086800       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:22:15.086840       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:22:27.998401       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:22:27.998448       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:22:28.768885       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:22:28.768926       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:22:29.123612       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:22:29.123704       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:22:29.124128       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:22:29.124158       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:22:29.124174       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:22:29.124189       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:22:29.124369       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:22:29.124631       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:22:29.124717       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:22:29.124758       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:22:29.124781       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:22:29.124801       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:22:29.124822       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:22:29.124836       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:22:29.124851       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:22:35.367334       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:22:35.367377       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:22:35.767498       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:22:35.767536       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:22:37.029034       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:22:37.029071       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:22:37.820356       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:22:37.820410       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:22:41.908270       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:22:41.908314       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:22:44.136685       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:22:44.136755       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:22:44.136776       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:22:44.137058       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:22:44.137180       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:22:44.137203       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:22:44.137247       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:22:44.137279       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:22:44.137300       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:22:44.137378       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:22:44.137405       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:22:44.137425       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:22:44.137442       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:22:44.137465       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:22:44.137481       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:22:45.970711       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:22:45.970758       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:22:53.527689       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:22:53.527735       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:22:57.636972       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:22:57.637016       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:22:59.143095       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:22:59.143193       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:22:59.143620       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:22:59.143641       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:22:59.143665       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:22:59.143681       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:22:59.143697       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:22:59.143711       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:22:59.143726       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:22:59.143809       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:22:59.143842       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:22:59.143862       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:22:59.143927       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:22:59.143956       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:22:59.143972       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:23:12.713433       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:23:12.713489       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:23:14.154744       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:23:14.154812       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:23:14.154837       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:23:14.155334       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:23:14.155370       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:23:14.155407       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:23:14.155466       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:23:14.155470       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:23:14.155529       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:23:14.155544       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:23:14.155550       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:23:14.155581       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:23:14.155604       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:23:14.155618       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:23:14.155633       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:23:17.458891       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:23:17.458930       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:23:19.713513       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:23:19.713558       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:23:22.292449       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:23:22.292497       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:23:24.474128       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:23:24.474170       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:23:24.506179       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:23:24.506205       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:23:25.694212       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:23:25.694256       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:23:29.026725       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:23:29.026785       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:23:29.159173       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:23:29.159582       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:23:29.159607       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:23:29.159843       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:23:29.159943       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:23:29.159964       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:23:29.160800       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:23:29.161022       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:23:29.161046       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:23:29.161223       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:23:29.161476       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:23:29.161501       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:23:29.161533       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:23:29.161642       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:23:29.161675       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:23:31.834345       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:23:31.834409       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:23:38.127183       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:23:38.127259       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:23:39.567878       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:23:39.567931       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:23:44.172968       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:23:44.173030       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:23:44.173070       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:23:44.173085       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:23:44.173095       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:23:44.172970       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:23:44.173127       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:23:44.173153       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:23:44.173167       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:23:44.173183       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:23:44.173199       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:23:44.173215       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:23:44.173304       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:23:44.173353       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:23:44.173377       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:23:47.191714       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:23:47.191755       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:23:52.511724       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:23:52.511771       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:23:57.062864       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:23:57.062916       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:23:59.178972       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:23:59.179012       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:23:59.179059       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:23:59.179080       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:23:59.179095       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:23:59.179127       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:23:59.179466       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:23:59.179495       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:23:59.179513       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:23:59.179603       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:23:59.179660       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:23:59.179662       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:23:59.179701       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:23:59.179742       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:23:59.179773       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:24:02.446016       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:24:02.446057       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:24:05.250453       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:24:05.250491       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:24:08.150456       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:24:08.150503       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:24:10.587346       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:24:10.587414       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:24:11.323517       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:24:11.323558       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:24:12.129467       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:24:12.129515       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:24:14.191534       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:24:14.191631       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:24:14.191693       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:24:14.191942       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:24:14.191977       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:24:14.191993       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:24:14.192010       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:24:14.192026       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:24:14.192041       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:24:14.192162       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:24:14.192186       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:24:14.192220       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:24:14.192242       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:24:14.192261       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:24:14.192277       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:24:18.396483       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:24:18.396530       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:24:27.537738       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:24:27.537787       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:24:29.198551       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:24:29.198777       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:24:29.198804       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:24:29.199014       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:24:29.199046       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:24:29.199072       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:24:29.199115       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:24:29.199141       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:24:29.199156       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:24:29.199159       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:24:29.199234       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:24:29.199276       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:24:29.199306       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:24:29.199340       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:24:29.199373       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:24:41.060856       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:24:41.060901       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:24:43.055381       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:24:43.055462       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:24:44.209559       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:24:44.209629       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:24:44.209652       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:24:44.210626       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:24:44.210630       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:24:44.210632       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:24:44.210682       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:24:44.210699       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:24:44.210716       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:24:44.210732       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:24:44.210746       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:24:44.210762       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:24:44.210765       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:24:44.210799       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:24:44.210821       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:24:45.156692       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:24:45.156734       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:24:46.020080       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:24:46.020127       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:24:56.207221       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:24:56.207273       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:24:58.090288       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:24:58.090328       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:24:58.957955       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:24:58.957999       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:24:59.215231       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:24:59.215249       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:24:59.215535       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:24:59.215563       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:24:59.215579       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:24:59.215595       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:24:59.215809       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:24:59.215885       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:24:59.215907       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:24:59.216233       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:24:59.216265       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:24:59.216283       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:24:59.216499       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:24:59.216536       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:24:59.216557       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:25:03.310475       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:25:03.310520       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:25:08.934633       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:25:08.934690       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:25:11.889322       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:25:11.889405       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:25:14.220345       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:25:14.220929       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:25:14.220959       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:25:14.221889       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:25:14.222143       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:25:14.222168       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:25:14.222356       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:25:14.222458       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:25:14.222476       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:25:14.222485       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:25:14.222540       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:25:14.222570       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:25:14.223438       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:25:14.223532       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:25:14.223569       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:25:21.601836       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:25:21.601893       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:25:25.063463       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:25:25.063504       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:25:29.226663       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:25:29.226764       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:25:29.227069       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:25:29.227092       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:25:29.227129       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:25:29.227146       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:25:29.227549       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:25:29.227578       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:25:29.227595       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:25:29.228297       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:25:29.228378       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:25:29.228437       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:25:29.229127       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:25:29.229359       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:25:29.229405       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:25:32.372056       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:25:32.372111       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:25:33.698530       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:25:33.698608       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:25:41.341928       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:25:41.341970       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:25:43.968741       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:25:43.968778       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:25:44.240180       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:25:44.240309       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:25:44.240412       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:25:44.240478       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:25:44.240518       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:25:44.240554       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:25:44.240574       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:25:44.240590       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:25:44.240606       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:25:44.241266       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:25:44.241366       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:25:44.241731       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:25:44.241759       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:25:44.241774       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:25:44.241790       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:25:46.774528       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:25:46.774569       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:25:55.768450       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:25:55.768500       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:25:57.759758       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:25:57.759802       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:25:59.247730       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:25:59.247828       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:25:59.247870       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:25:59.247890       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:25:59.247905       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:25:59.247936       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:25:59.247940       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:25:59.247986       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:25:59.248021       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:25:59.248075       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:25:59.248115       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:25:59.248141       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:25:59.248147       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:25:59.248187       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:25:59.248220       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:26:00.280313       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:26:00.280354       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:26:00.888480       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:26:00.888525       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:26:01.788995       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:26:01.789038       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:26:07.415568       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:26:07.415612       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:26:13.487824       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:26:13.487862       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:26:14.265322       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:26:14.265446       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:26:14.265526       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:26:14.265324       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:26:14.265598       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:26:14.265655       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:26:14.265689       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:26:14.265719       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:26:14.265753       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:26:14.265776       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:26:14.265800       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:26:14.265829       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:26:14.265853       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:26:14.265882       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:26:14.265912       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:26:14.769821       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:26:14.769868       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:26:15.431422       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:26:15.431468       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:26:15.804152       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:26:15.804194       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:26:29.270257       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:26:29.270494       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:26:29.270522       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:26:29.271288       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:26:29.271592       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:26:29.271606       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:26:29.271614       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:26:29.271631       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:26:29.271683       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:26:29.271698       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:26:29.271702       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:26:29.271743       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:26:29.271768       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:26:29.271790       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:26:29.271808       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:26:31.628663       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:26:31.628711       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:26:44.281381       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:26:44.281665       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:26:44.281688       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:26:44.281872       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:26:44.281905       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:26:44.281938       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:26:44.281959       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:26:44.281973       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:26:44.281990       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:26:44.282659       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:26:44.282816       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:26:44.282857       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:26:44.282879       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:26:44.282894       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:26:44.282915       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:26:44.478023       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:26:44.478065       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:26:45.163542       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:26:45.163578       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:26:49.956370       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:26:49.956425       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:26:51.407333       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:26:51.407371       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:26:55.903863       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:26:55.903903       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:26:57.347926       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:26:57.347968       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:26:57.748051       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:26:57.748088       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:26:59.287334       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:26:59.287422       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:26:59.287447       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:26:59.287740       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:26:59.287752       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:26:59.287910       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:26:59.287967       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:26:59.287997       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:26:59.288014       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:26:59.288030       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:26:59.288050       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:26:59.288067       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:26:59.288083       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:26:59.288110       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:26:59.288128       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:27:03.293659       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:27:03.293698       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:27:07.411152       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:27:07.411194       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:27:11.747561       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:27:11.747602       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:27:12.316014       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:27:12.316063       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:27:14.299012       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:27:14.299070       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:27:14.299109       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:27:14.299910       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:27:14.300163       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:27:14.300185       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:27:14.300217       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:27:14.300284       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:27:14.300302       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:27:14.300306       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:27:14.300339       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:27:14.300359       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:27:14.300371       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:27:14.300420       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:27:14.300440       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:27:17.429624       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:27:17.429665       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:27:26.108961       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:27:26.109002       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:27:28.018032       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:27:28.018072       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:27:29.305341       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:27:29.305847       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:27:29.305884       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:27:29.305976       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:27:29.306301       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:27:29.306336       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:27:29.306587       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:27:29.306664       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:27:29.306691       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:27:29.307063       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:27:29.307143       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:27:29.307175       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:27:29.307194       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:27:29.307209       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:27:29.307224       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:27:32.106254       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:27:32.106311       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:27:32.882838       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:27:32.882879       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:27:35.558659       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:27:35.558697       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:27:35.608002       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:27:35.608030       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:27:39.898918       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:27:39.898958       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:27:40.434431       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:27:40.434467       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:27:44.318497       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:27:44.318608       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:27:44.318679       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:27:44.318779       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:27:44.319668       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:27:44.319711       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:27:44.319732       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:27:44.319748       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:27:44.319771       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:27:44.319807       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:27:44.319826       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:27:44.319847       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:27:44.320010       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:27:44.320245       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:27:44.320273       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:27:59.171013       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:27:59.171102       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:27:59.324937       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:27:59.325067       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:27:59.325526       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:27:59.325559       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:27:59.325575       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:27:59.325593       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:27:59.325686       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:27:59.325749       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:27:59.325849       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:27:59.325901       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:27:59.325937       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:27:59.325957       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:27:59.325973       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:27:59.325988       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:27:59.326003       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:28:05.344589       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:28:05.344638       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:28:09.432581       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:28:09.432627       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:28:09.623296       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:28:09.623356       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:28:12.139794       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:28:12.139838       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:28:13.698889       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:28:13.698934       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:28:14.337664       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:28:14.337735       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:28:14.337782       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:28:14.337805       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:28:14.337823       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:28:14.337843       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:28:14.338593       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:28:14.338597       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:28:14.338601       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:28:14.338624       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:28:14.338647       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:28:14.338663       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:28:14.338680       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:28:14.338696       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:28:14.338712       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:28:19.096107       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:28:19.096159       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:28:25.174775       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:28:25.174830       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:28:27.086276       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:28:27.086316       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:28:29.343797       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:28:29.344067       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:28:29.344109       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:28:29.344277       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:28:29.344292       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:28:29.344357       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:28:29.344499       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:28:29.344494       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:28:29.344536       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:28:29.344558       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:28:29.344574       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:28:29.344587       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:28:29.344602       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:28:29.344616       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:28:29.344630       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:28:30.336796       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:28:30.336838       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:28:30.958598       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:28:30.958636       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:28:35.070950       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:28:35.070994       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:28:44.355634       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:28:44.355696       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:28:44.355701       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:28:44.355750       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:28:44.355768       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:28:44.355788       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:28:44.355922       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:28:44.355951       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:28:44.355973       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:28:44.356116       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:28:44.356150       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:28:44.356170       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:28:44.356297       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:28:44.356331       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:28:44.356348       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:28:44.996272       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:28:44.996342       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:28:46.580674       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:28:46.580719       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:28:59.361380       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:28:59.361568       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:28:59.361597       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:28:59.362703       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:28:59.362763       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:28:59.362792       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:28:59.362918       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:28:59.362956       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:28:59.362973       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:28:59.363037       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:28:59.363086       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:28:59.363124       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:28:59.363211       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:28:59.363248       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:28:59.363268       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:29:04.283654       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:29:04.283707       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:29:06.209726       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:29:06.209766       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:29:06.663357       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:29:06.663421       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:29:06.707227       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:29:06.707282       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:29:07.751514       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:29:07.751563       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:29:11.101890       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:29:11.101936       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:29:11.339020       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:29:11.339063       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:29:14.373697       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:29:14.373703       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:29:14.373735       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:29:14.373757       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:29:14.373775       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:29:14.373792       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:29:14.374073       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:29:14.374113       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:29:14.374132       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:29:14.374191       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:29:14.374223       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:29:14.374240       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:29:14.374363       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:29:14.374418       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:29:14.374442       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:29:16.280218       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:29:16.280274       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:29:23.327213       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:29:23.327271       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:29:25.507903       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:29:25.507948       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:29:29.379148       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:29:29.379475       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:29:29.379504       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:29:29.379880       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:29:29.380119       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:29:29.380147       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:29:29.380594       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:29:29.380611       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:29:29.380639       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:29:29.380682       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:29:29.380705       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:29:29.380723       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:29:29.381129       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:29:29.381179       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:29:29.381208       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:29:41.395033       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:29:41.395103       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:29:44.391773       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:29:44.391868       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:29:44.391927       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:29:44.392144       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:29:44.392266       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:29:44.392310       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:29:44.392330       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:29:44.392345       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:29:44.392360       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:29:44.392374       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:29:44.392411       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:29:44.392427       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:29:44.392443       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:29:44.392457       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:29:44.392472       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:29:44.740460       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:29:44.740502       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:29:48.279869       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:29:48.279909       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:29:50.777691       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:29:50.777732       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:29:51.198471       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:29:51.198509       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:29:56.989101       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:29:56.989143       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:29:57.047517       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:29:57.047541       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:29:57.181057       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:29:57.181108       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:29:59.396494       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:29:59.396578       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:29:59.396602       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:29:59.397496       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:29:59.397561       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:29:59.397581       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:29:59.398098       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:29:59.398331       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:29:59.398554       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:29:59.398586       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:29:59.398602       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:29:59.398618       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:29:59.398632       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:29:59.398648       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:29:59.398565       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
W0508 06:30:06.104207       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:30:06.104246       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:30:12.879627       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:30:12.879665       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:30:14.401530       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:30:14.401804       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:30:14.401836       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:30:14.402299       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:30:14.402566       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:30:14.402634       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:30:14.402672       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:30:14.402784       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:30:14.402814       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:30:14.403793       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:30:14.403892       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:30:14.403918       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:30:14.404075       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:30:14.404155       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:30:14.404177       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:30:15.483074       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:30:15.483140       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:30:22.850256       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:30:22.850303       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:30:29.406064       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:30:29.406365       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:30:29.406414       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:30:29.406473       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:30:29.406655       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:30:29.406684       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:30:29.407528       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:30:29.407607       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:30:29.407642       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:30:29.407661       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:30:29.407676       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:30:29.407692       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:30:29.408955       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:30:29.408997       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:30:29.409015       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:30:33.506753       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:30:33.506796       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:30:33.738139       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:30:33.738187       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:30:43.398196       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:30:43.398237       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:30:43.947911       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:30:43.947952       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:30:44.416818       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:30:44.417119       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:30:44.417151       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:30:44.417275       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:30:44.417303       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:30:44.417321       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:30:44.417726       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:30:44.417741       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:30:44.417880       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:30:44.417924       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:30:44.417942       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:30:44.417961       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:30:44.417977       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:30:44.417991       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:30:44.418005       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:30:44.938886       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:30:44.938926       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:30:47.736838       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:30:47.736879       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:30:50.414490       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:30:50.414535       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:30:50.992349       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:30:50.992410       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:30:55.341521       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:30:55.341560       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:30:58.328011       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:30:58.328050       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:30:59.422702       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:30:59.422738       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:30:59.423026       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:30:59.423053       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:30:59.423071       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:30:59.423099       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:30:59.423903       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:30:59.423922       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:30:59.423952       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:30:59.423969       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:30:59.423984       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:30:59.424000       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:30:59.424324       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:30:59.424415       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:30:59.424444       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:31:11.702976       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:31:11.703029       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:31:14.343862       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:31:14.343906       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:31:14.433689       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:31:14.433782       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:31:14.434189       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:31:14.434215       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:31:14.434231       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:31:14.434247       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:31:14.434436       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:31:14.434463       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:31:14.434482       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:31:14.434556       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:31:14.434582       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:31:14.434600       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:31:14.434661       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:31:14.434683       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:31:14.434699       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:31:18.890932       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:31:18.890973       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:31:21.981553       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:31:21.981606       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:31:22.689742       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:31:22.689783       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:31:23.233305       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:31:23.233359       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:31:29.441958       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:31:29.442102       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:31:29.442138       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:31:29.442259       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:31:29.442305       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:31:29.442325       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:31:29.442340       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:31:29.442355       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:31:29.442369       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:31:29.442410       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:31:29.442428       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:31:29.442444       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:31:29.444038       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:31:29.444112       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:31:29.444147       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:31:31.712703       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:31:31.712743       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:31:32.372011       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:31:32.372052       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:31:41.781781       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:31:41.781823       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:31:42.967973       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:31:42.968018       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:31:44.455790       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:31:44.455884       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:31:44.455973       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:31:44.456017       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:31:44.456064       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:31:44.456095       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:31:44.456119       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:31:44.456134       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:31:44.456150       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:31:44.456415       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:31:44.456442       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:31:44.456460       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:31:44.456587       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:31:44.456623       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:31:44.456645       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:31:47.642896       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:31:47.642942       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:31:51.312441       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:31:51.312479       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:31:58.537346       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:31:58.537402       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:31:59.171802       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:31:59.171839       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:31:59.461646       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:31:59.462150       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:31:59.462180       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:31:59.464227       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:31:59.464257       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:31:59.464267       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:31:59.464307       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:31:59.464317       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:31:59.464330       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:31:59.464346       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:31:59.464362       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:31:59.464378       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:31:59.464427       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:31:59.464455       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:31:59.464490       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:32:03.463168       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:32:03.463212       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:32:09.447650       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:32:09.447710       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:32:12.192552       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:32:12.192594       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:32:14.476529       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:32:14.476612       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:32:14.477001       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:32:14.477033       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:32:14.477049       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:32:14.477065       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:32:14.477260       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:32:14.477536       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:32:14.477624       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:32:14.477661       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:32:14.477683       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:32:14.477698       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:32:14.477713       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:32:14.477902       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:32:14.477927       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:32:20.034622       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:32:20.034667       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:32:21.570467       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:32:21.570505       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:32:29.482404       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:32:29.482474       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:32:29.482473       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:32:29.482561       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:32:29.482602       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:32:29.482630       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:32:29.482655       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:32:29.482682       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:32:29.482708       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:32:29.483767       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:32:29.483863       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:32:29.483903       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:32:29.484567       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:32:29.484633       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:32:29.484654       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:32:31.954715       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:32:31.954764       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:32:35.886649       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:32:35.886692       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:32:39.923329       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:32:39.923378       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:32:43.496791       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:32:43.496852       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:32:44.495300       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:32:44.495379       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:32:44.495801       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:32:44.495840       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:32:44.495867       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:32:44.495892       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:32:44.495924       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:32:44.496021       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:32:44.496049       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:32:44.496238       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:32:44.496366       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:32:44.496426       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:32:44.496546       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:32:44.496600       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:32:44.496626       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:32:44.871125       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:32:44.871171       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:32:46.181783       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:32:46.181827       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:32:51.451786       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:32:51.451837       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:32:54.482994       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:32:54.483042       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:32:57.433379       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:32:57.433442       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:32:59.501115       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:32:59.501224       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:32:59.501242       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:32:59.501419       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:32:59.501426       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:32:59.501446       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:32:59.501461       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:32:59.501478       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:32:59.501526       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:32:59.501556       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:32:59.501571       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:32:59.501586       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:32:59.502171       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:32:59.502208       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:32:59.502227       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:33:10.006870       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:33:10.006936       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:33:14.514571       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:33:14.514991       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:33:14.515017       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:33:14.515028       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:33:14.515047       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:33:14.515090       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:33:14.515093       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:33:14.515132       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:33:14.515150       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:33:14.515472       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:33:14.515515       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:33:14.515521       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:33:14.515537       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:33:14.515553       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:33:14.515577       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:33:16.876879       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:33:16.876928       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:33:19.831722       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:33:19.831775       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:33:22.877851       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:33:22.877914       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:33:24.213834       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:33:24.213928       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:33:26.983559       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:33:26.983600       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:33:29.527206       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:33:29.527206       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:33:29.527242       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:33:29.527272       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:33:29.527281       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:33:29.527306       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:33:29.527327       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:33:29.527345       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:33:29.527363       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:33:29.527381       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:33:29.527427       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:33:29.527444       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:33:29.527459       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:33:29.527474       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:33:29.527489       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:33:30.695518       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:33:30.695564       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:33:31.676665       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:33:31.676706       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:33:41.065805       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:33:41.065866       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:33:43.616520       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:33:43.616564       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:33:44.532967       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:33:44.533472       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:33:44.533501       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:33:44.533649       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:33:44.533900       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:33:44.534155       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:33:44.534340       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:33:44.534368       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:33:44.534383       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:33:44.534418       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:33:44.534431       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:33:44.534448       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:33:44.534580       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:33:44.534783       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:33:44.534809       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:33:48.489304       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:33:48.489355       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:33:52.262521       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:33:52.262569       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:33:58.481808       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:33:58.481848       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:33:59.543285       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:33:59.543406       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:33:59.543449       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:33:59.544338       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:33:59.544699       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:33:59.544726       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:33:59.544751       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:33:59.544768       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:33:59.544783       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:33:59.545741       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:33:59.545753       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:33:59.545789       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:33:59.545808       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:33:59.545824       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:33:59.545841       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:34:02.963663       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:34:02.963716       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:34:08.059897       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:34:08.059936       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:34:14.547688       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:34:14.547978       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:34:14.548003       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:34:14.548359       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:34:14.548591       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:34:14.548613       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:34:14.549472       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:34:14.549627       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:34:14.549652       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:34:14.549813       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:34:14.549840       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:34:14.549857       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:34:14.550145       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:34:14.550177       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:34:14.550197       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:34:15.484382       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:34:15.484441       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:34:15.488521       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:34:15.488552       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:34:18.640808       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:34:18.640848       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:34:21.308517       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:34:21.308559       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:34:22.063537       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:34:22.063577       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:34:22.547669       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:34:22.547713       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:34:22.789802       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:34:22.789839       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:34:29.559024       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:34:29.559053       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:34:29.559151       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:34:29.559209       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:34:29.559251       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:34:29.559271       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:34:29.559288       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:34:29.559289       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:34:29.559331       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:34:29.559363       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:34:29.559379       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:34:29.559412       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:34:29.559427       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:34:29.559441       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:34:29.559455       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:34:35.302849       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:34:35.302890       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:34:42.841641       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:34:42.841685       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:34:42.897756       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:34:42.897782       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:34:44.564846       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:34:44.564931       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:34:44.564955       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:34:44.565300       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:34:44.565342       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:34:44.565374       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:34:44.565412       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:34:44.565432       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:34:44.565447       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:34:44.565587       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:34:44.565614       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:34:44.565633       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:34:44.565786       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:34:44.565823       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:34:44.565842       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:34:56.104637       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:34:56.104677       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:34:58.594227       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:34:58.594273       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:34:59.406610       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:34:59.406654       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:34:59.577295       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:34:59.577335       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:34:59.577419       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:34:59.577427       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:34:59.577419       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:34:59.577456       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:34:59.577477       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:34:59.577499       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:34:59.577517       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:34:59.577542       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:34:59.577557       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:34:59.577571       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:34:59.577587       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:34:59.577609       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:34:59.577624       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:35:01.482869       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:35:01.482933       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:35:03.026292       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:35:03.026332       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:35:12.134566       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:35:12.134607       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:35:14.166766       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:35:14.166813       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:35:14.584173       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:35:14.584519       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:35:14.584543       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:35:14.584757       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:35:14.584889       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:35:14.584977       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:35:14.585056       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:35:14.585104       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:35:14.585130       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:35:14.585149       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:35:14.585168       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:35:14.585185       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:35:14.585214       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:35:14.585231       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:35:14.585248       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:35:18.244813       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:35:18.244869       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:35:20.277128       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:35:20.277172       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:35:29.597462       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:35:29.597606       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:35:29.597807       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:35:29.597890       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:35:29.598051       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:35:29.598121       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:35:29.598149       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:35:29.598168       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:35:29.598186       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:35:29.598202       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:35:29.598224       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:35:29.598240       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:35:29.598261       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:35:29.598285       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:35:29.598303       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:35:31.312219       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:35:31.312269       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:35:38.129803       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:35:38.129842       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:35:38.860293       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:35:38.860338       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:35:40.938087       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:35:40.938132       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:35:41.518024       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:35:41.518061       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:35:44.602603       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:35:44.602846       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:35:44.602874       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:35:44.603952       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:35:44.604203       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:35:44.604243       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:35:44.604352       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:35:44.604420       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:35:44.604443       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:35:44.605864       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:35:44.606135       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:35:44.606186       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:35:44.606349       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:35:44.606607       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:35:44.606647       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:35:46.289348       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:35:46.289411       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:35:50.342922       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:35:50.342960       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:35:55.431655       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:35:55.431704       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:35:57.092956       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:35:57.092993       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:35:59.614671       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:35:59.614707       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:35:59.614732       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:35:59.614769       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:35:59.614803       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:35:59.614828       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:35:59.615816       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:35:59.615829       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:35:59.615881       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:35:59.615901       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:35:59.615917       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:35:59.615933       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:35:59.616154       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:35:59.616181       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:35:59.616197       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:36:02.169048       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:36:02.169127       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:36:06.430031       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:36:06.430087       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:36:14.619506       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:36:14.619821       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:36:14.619845       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:36:14.620846       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:36:14.620977       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:36:14.621023       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:36:14.621120       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:36:14.621159       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:36:14.621183       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:36:14.621193       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:36:14.621198       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:36:14.621246       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
E0508 06:36:14.621666       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:36:14.621712       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:36:14.621733       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:36:15.213149       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:36:15.213189       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:36:17.635635       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:36:17.635687       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:36:19.305513       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:36:19.305556       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:36:27.591592       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:36:27.591634       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:36:29.631602       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istiod: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:36:29.631682       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:36:29.631851       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:36:29.631936       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/cluster-local-gateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
E0508 06:36:29.632065       1 horizontal.go:274] failed to compute desired number of replicas based on listed metrics for Deployment/istio-system/istio-ingressgateway: invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
I0508 06:36:29.632126       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:36:29.632148       1 event.go:376] "Event occurred" object="istio-system/istiod" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:36:29.632165       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:36:29.632180       1 event.go:376] "Event occurred" object="knative-serving/webhook" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:36:29.632195       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:36:29.632211       1 event.go:376] "Event occurred" object="knative-serving/activator" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:36:29.632228       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:36:29.632244       1 event.go:376] "Event occurred" object="istio-system/cluster-local-gateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:36:29.632259       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
I0508 06:36:29.632275       1 event.go:376] "Event occurred" object="istio-system/istio-ingressgateway" fieldPath="" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)"
W0508 06:36:30.022263       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:36:30.022311       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
W0508 06:36:30.782354       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource
E0508 06:36:30.782425       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: the server could not find the requested resource

Pod: kube-proxy-j9hkr (Namespace: kube-system)

Logs (filtered)

W0425 06:37:46.321337       1 reflector.go:462] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: watch of *v1.Service ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0425 06:37:46.374237       1 reflector.go:462] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0425 06:37:46.422103       1 reflector.go:462] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: watch of *v1.EndpointSlice ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0425 06:38:53.379255       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90233": net/http: TLS handshake timeout
W0425 06:38:53.379346       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90225": net/http: TLS handshake timeout
Trace[3554336]: ---"Objects listed" error:Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90233": net/http: TLS handshake timeout 62563ms (06:38:52.086)
Trace[1383159587]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90225": net/http: TLS handshake timeout 62421ms (06:38:52.086)
E0425 06:39:04.126478       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90233": net/http: TLS handshake timeout
E0425 06:39:04.640550       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90225": net/http: TLS handshake timeout
W0425 06:39:04.602697       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-master01&resourceVersion=90296": net/http: TLS handshake timeout
Trace[201964045]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-master01&resourceVersion=90296": net/http: TLS handshake timeout 75058ms (06:39:04.583)
E0425 06:39:30.239506       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-master01&resourceVersion=90296": net/http: TLS handshake timeout
W0425 06:40:54.046770       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90233": net/http: TLS handshake timeout
W0425 06:40:54.046473       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90225": net/http: TLS handshake timeout
Trace[76351691]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90225": net/http: TLS handshake timeout 104975ms (06:40:52.699)
Trace[794795268]: ---"Objects listed" error:Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90233": net/http: TLS handshake timeout 104561ms (06:40:52.699)
E0425 06:41:17.362947       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90233": net/http: TLS handshake timeout
E0425 06:41:21.378630       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90225": net/http: TLS handshake timeout
W0425 06:42:23.349732       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-master01&resourceVersion=90296": net/http: TLS handshake timeout
Trace[102776452]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-master01&resourceVersion=90296": net/http: TLS handshake timeout 166636ms (06:42:23.023)
E0425 06:42:25.229373       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-master01&resourceVersion=90296": net/http: TLS handshake timeout
Trace[124830391]: ---"Objects listed" error: 60794ms (06:42:26.267)
Trace[1360826250]: ---"Objects listed" error: 62825ms (06:42:26.270)

Pod: kube-proxy-m4dwc (Namespace: kube-system)

Logs (filtered)

W0425 06:37:45.767751       1 reflector.go:462] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: watch of *v1.Service ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0425 06:37:45.767812       1 reflector.go:462] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: watch of *v1.EndpointSlice ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0425 06:37:45.767750       1 reflector.go:462] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0425 06:37:56.590800       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90314": net/http: TLS handshake timeout
Trace[318960195]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90314": net/http: TLS handshake timeout 10005ms (06:37:56.590)
E0425 06:37:56.590953       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90314": net/http: TLS handshake timeout
W0425 06:37:56.613864       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90287": net/http: TLS handshake timeout
Trace[1716165448]: ---"Objects listed" error:Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90287": net/http: TLS handshake timeout 10001ms (06:37:56.613)
E0425 06:37:56.613942       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90287": net/http: TLS handshake timeout
W0425 06:37:56.851306       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker01&resourceVersion=90230": net/http: TLS handshake timeout
Trace[611101159]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker01&resourceVersion=90230": net/http: TLS handshake timeout 10001ms (06:37:56.851)
E0425 06:37:56.851377       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker01&resourceVersion=90230": net/http: TLS handshake timeout
W0425 06:38:08.274378       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90287": net/http: TLS handshake timeout
Trace[826341723]: ---"Objects listed" error:Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90287": net/http: TLS handshake timeout 10001ms (06:38:08.274)
E0425 06:38:08.274472       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90287": net/http: TLS handshake timeout
W0425 06:38:09.034261       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90314": net/http: TLS handshake timeout
Trace[281973341]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90314": net/http: TLS handshake timeout 10001ms (06:38:09.034)
E0425 06:38:09.034344       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90314": net/http: TLS handshake timeout
W0425 06:38:09.303661       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker01&resourceVersion=90230": net/http: TLS handshake timeout
Trace[327050551]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker01&resourceVersion=90230": net/http: TLS handshake timeout 10001ms (06:38:09.303)
E0425 06:38:09.303744       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker01&resourceVersion=90230": net/http: TLS handshake timeout
W0425 06:38:22.555993       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker01&resourceVersion=90230": net/http: TLS handshake timeout
Trace[1385292269]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker01&resourceVersion=90230": net/http: TLS handshake timeout 10001ms (06:38:22.555)
E0425 06:38:22.556079       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker01&resourceVersion=90230": net/http: TLS handshake timeout
W0425 06:38:22.896393       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90314": net/http: TLS handshake timeout
Trace[1021637456]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90314": net/http: TLS handshake timeout 10001ms (06:38:22.896)
E0425 06:38:22.896500       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90314": net/http: TLS handshake timeout
W0425 06:38:23.039701       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90287": net/http: TLS handshake timeout
Trace[1639532465]: ---"Objects listed" error:Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90287": net/http: TLS handshake timeout 10001ms (06:38:23.039)
E0425 06:38:23.039782       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90287": net/http: TLS handshake timeout
W0425 06:38:41.105287       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90287": net/http: TLS handshake timeout
Trace[1116399865]: ---"Objects listed" error:Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90287": net/http: TLS handshake timeout 10004ms (06:38:41.105)
E0425 06:38:41.105375       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90287": net/http: TLS handshake timeout
W0425 06:38:42.246201       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker01&resourceVersion=90230": net/http: TLS handshake timeout
Trace[1795730859]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker01&resourceVersion=90230": net/http: TLS handshake timeout 10001ms (06:38:42.246)
E0425 06:38:42.246275       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker01&resourceVersion=90230": net/http: TLS handshake timeout
W0425 06:38:42.495568       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90314": net/http: TLS handshake timeout
Trace[1868432343]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90314": net/http: TLS handshake timeout 10000ms (06:38:42.495)
E0425 06:38:42.495646       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90314": net/http: TLS handshake timeout
W0425 06:39:09.746435       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker01&resourceVersion=90230": net/http: TLS handshake timeout
Trace[1596074481]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker01&resourceVersion=90230": net/http: TLS handshake timeout 10002ms (06:39:09.746)
E0425 06:39:09.746557       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker01&resourceVersion=90230": net/http: TLS handshake timeout
W0425 06:39:10.175994       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90287": net/http: TLS handshake timeout
Trace[1712012262]: ---"Objects listed" error:Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90287": net/http: TLS handshake timeout 10000ms (06:39:10.175)
E0425 06:39:10.176107       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90287": net/http: TLS handshake timeout
W0425 06:39:15.398273       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90314": net/http: TLS handshake timeout
Trace[214287947]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90314": net/http: TLS handshake timeout 10004ms (06:39:15.398)
E0425 06:39:15.398386       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90314": net/http: TLS handshake timeout
W0425 06:39:53.413357       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker01&resourceVersion=90230": net/http: TLS handshake timeout
Trace[1897281553]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker01&resourceVersion=90230": net/http: TLS handshake timeout 10001ms (06:39:53.413)
E0425 06:39:53.413464       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker01&resourceVersion=90230": net/http: TLS handshake timeout
W0425 06:39:54.645651       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90287": net/http: TLS handshake timeout
Trace[263631961]: ---"Objects listed" error:Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90287": net/http: TLS handshake timeout 10001ms (06:39:54.645)
E0425 06:39:54.645754       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90287": net/http: TLS handshake timeout
W0425 06:39:58.389703       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90314": net/http: TLS handshake timeout
Trace[620441770]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90314": net/http: TLS handshake timeout 10002ms (06:39:58.389)
E0425 06:39:58.389836       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90314": net/http: TLS handshake timeout
W0425 06:40:40.700845       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90287": net/http: TLS handshake timeout
Trace[1850515236]: ---"Objects listed" error:Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90287": net/http: TLS handshake timeout 10003ms (06:40:40.700)
E0425 06:40:40.700972       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90287": net/http: TLS handshake timeout
W0425 06:40:42.431814       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90314": net/http: TLS handshake timeout
Trace[622874390]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90314": net/http: TLS handshake timeout 10001ms (06:40:42.431)
E0425 06:40:42.431896       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90314": net/http: TLS handshake timeout
W0425 06:40:54.932253       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker01&resourceVersion=90230": net/http: TLS handshake timeout
Trace[403575321]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker01&resourceVersion=90230": net/http: TLS handshake timeout 10001ms (06:40:54.932)
E0425 06:40:54.932347       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker01&resourceVersion=90230": net/http: TLS handshake timeout
W0425 06:41:28.672390       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90314": net/http: TLS handshake timeout
Trace[1238649649]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90314": net/http: TLS handshake timeout 10001ms (06:41:28.672)
E0425 06:41:28.672507       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90314": net/http: TLS handshake timeout
W0425 06:41:31.512106       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90287": net/http: TLS handshake timeout
Trace[1433389425]: ---"Objects listed" error:Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90287": net/http: TLS handshake timeout 10001ms (06:41:31.512)
E0425 06:41:31.512251       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90287": net/http: TLS handshake timeout
W0425 06:41:59.172328       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker01&resourceVersion=90230": net/http: TLS handshake timeout
Trace[1340810292]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker01&resourceVersion=90230": net/http: TLS handshake timeout 10002ms (06:41:59.172)
E0425 06:41:59.172427       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker01&resourceVersion=90230": net/http: TLS handshake timeout

Pod: kube-proxy-t7hf5 (Namespace: kube-system)

Logs (filtered)

W0425 06:37:35.390155       1 reflector.go:462] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0425 06:37:35.390159       1 reflector.go:462] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: watch of *v1.Service ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0425 06:37:35.390150       1 reflector.go:462] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: watch of *v1.EndpointSlice ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0425 06:37:46.264802       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90164": net/http: TLS handshake timeout
Trace[776405900]: ---"Objects listed" error:Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90164": net/http: TLS handshake timeout 10001ms (06:37:46.264)
E0425 06:37:46.265048       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90164": net/http: TLS handshake timeout
W0425 06:37:46.487974       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker02&resourceVersion=90280": net/http: TLS handshake timeout
Trace[743702958]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker02&resourceVersion=90280": net/http: TLS handshake timeout 10001ms (06:37:46.487)
E0425 06:37:46.488079       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker02&resourceVersion=90280": net/http: TLS handshake timeout
W0425 06:37:46.872518       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90243": net/http: TLS handshake timeout
Trace[1951585778]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90243": net/http: TLS handshake timeout 10001ms (06:37:46.872)
E0425 06:37:46.872644       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90243": net/http: TLS handshake timeout
W0425 06:37:58.981589       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90164": net/http: TLS handshake timeout
Trace[1372933272]: ---"Objects listed" error:Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90164": net/http: TLS handshake timeout 10004ms (06:37:58.981)
E0425 06:37:58.981706       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90164": net/http: TLS handshake timeout
W0425 06:37:59.263964       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90243": net/http: TLS handshake timeout
Trace[891366288]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90243": net/http: TLS handshake timeout 10001ms (06:37:59.263)
E0425 06:37:59.264069       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90243": net/http: TLS handshake timeout
W0425 06:37:59.473329       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker02&resourceVersion=90280": net/http: TLS handshake timeout
Trace[1513851246]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker02&resourceVersion=90280": net/http: TLS handshake timeout 10001ms (06:37:59.473)
E0425 06:37:59.473487       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker02&resourceVersion=90280": net/http: TLS handshake timeout
W0425 06:38:12.727711       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker02&resourceVersion=90280": net/http: TLS handshake timeout
Trace[605512489]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker02&resourceVersion=90280": net/http: TLS handshake timeout 10001ms (06:38:12.727)
E0425 06:38:12.727824       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker02&resourceVersion=90280": net/http: TLS handshake timeout
W0425 06:38:12.989944       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90243": net/http: TLS handshake timeout
Trace[766420344]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90243": net/http: TLS handshake timeout 10001ms (06:38:12.989)
E0425 06:38:12.990047       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90243": net/http: TLS handshake timeout
W0425 06:38:14.550231       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90164": net/http: TLS handshake timeout
Trace[1265769882]: ---"Objects listed" error:Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90164": net/http: TLS handshake timeout 10001ms (06:38:14.550)
E0425 06:38:14.550373       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90164": net/http: TLS handshake timeout
W0425 06:38:31.213676       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90243": net/http: TLS handshake timeout
Trace[447593669]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90243": net/http: TLS handshake timeout 10001ms (06:38:31.213)
E0425 06:38:31.213793       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90243": net/http: TLS handshake timeout
W0425 06:38:33.655891       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90164": net/http: TLS handshake timeout
Trace[1182203723]: ---"Objects listed" error:Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90164": net/http: TLS handshake timeout 10001ms (06:38:33.655)
E0425 06:38:33.655994       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90164": net/http: TLS handshake timeout
W0425 06:38:34.703767       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker02&resourceVersion=90280": net/http: TLS handshake timeout
Trace[1511661630]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker02&resourceVersion=90280": net/http: TLS handshake timeout 10001ms (06:38:34.703)
E0425 06:38:34.703901       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker02&resourceVersion=90280": net/http: TLS handshake timeout
W0425 06:39:01.563797       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90243": net/http: TLS handshake timeout
Trace[2013947373]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90243": net/http: TLS handshake timeout 10002ms (06:39:01.563)
E0425 06:39:01.563907       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90243": net/http: TLS handshake timeout
W0425 06:39:02.276639       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker02&resourceVersion=90280": net/http: TLS handshake timeout
Trace[2054900420]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker02&resourceVersion=90280": net/http: TLS handshake timeout 10001ms (06:39:02.276)
E0425 06:39:02.276742       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker02&resourceVersion=90280": net/http: TLS handshake timeout
W0425 06:39:02.369900       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90164": net/http: TLS handshake timeout
Trace[1663988876]: ---"Objects listed" error:Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90164": net/http: TLS handshake timeout 10001ms (06:39:02.369)
E0425 06:39:02.370006       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90164": net/http: TLS handshake timeout
W0425 06:39:48.094000       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker02&resourceVersion=90280": net/http: TLS handshake timeout
Trace[1510861235]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker02&resourceVersion=90280": net/http: TLS handshake timeout 10003ms (06:39:48.093)
E0425 06:39:48.094099       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker02&resourceVersion=90280": net/http: TLS handshake timeout
W0425 06:39:48.236282       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90164": net/http: TLS handshake timeout
Trace[1765231135]: ---"Objects listed" error:Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90164": net/http: TLS handshake timeout 10001ms (06:39:48.236)
E0425 06:39:48.236385       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90164": net/http: TLS handshake timeout
W0425 06:39:59.230128       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90243": net/http: TLS handshake timeout
Trace[1087775280]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90243": net/http: TLS handshake timeout 10001ms (06:39:59.230)
E0425 06:39:59.230280       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90243": net/http: TLS handshake timeout
W0425 06:40:44.227632       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90243": net/http: TLS handshake timeout
Trace[298599866]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90243": net/http: TLS handshake timeout 10002ms (06:40:44.227)
E0425 06:40:44.227809       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90243": net/http: TLS handshake timeout
W0425 06:40:46.555894       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker02&resourceVersion=90280": net/http: TLS handshake timeout
Trace[867720724]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker02&resourceVersion=90280": net/http: TLS handshake timeout 10002ms (06:40:46.555)
E0425 06:40:46.556054       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker02&resourceVersion=90280": net/http: TLS handshake timeout
W0425 06:40:54.109667       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90164": net/http: TLS handshake timeout
Trace[648327222]: ---"Objects listed" error:Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90164": net/http: TLS handshake timeout 10004ms (06:40:54.109)
E0425 06:40:54.109802       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90164": net/http: TLS handshake timeout
W0425 06:41:29.131771       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90243": net/http: TLS handshake timeout
Trace[1338284438]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90243": net/http: TLS handshake timeout 10004ms (06:41:29.131)
E0425 06:41:29.131868       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90243": net/http: TLS handshake timeout
W0425 06:41:34.139904       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker02&resourceVersion=90280": net/http: TLS handshake timeout
Trace[1668996479]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker02&resourceVersion=90280": net/http: TLS handshake timeout 10004ms (06:41:34.139)
E0425 06:41:34.139992       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-kubeflow-worker02&resourceVersion=90280": net/http: TLS handshake timeout
W0425 06:41:55.876596       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90164": net/http: TLS handshake timeout
Trace[201211911]: ---"Objects listed" error:Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90164": net/http: TLS handshake timeout 10001ms (06:41:55.876)
E0425 06:41:55.876700       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://192.168.2.231:6443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90164": net/http: TLS handshake timeout
W0425 06:42:11.177108       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90243": net/http: TLS handshake timeout
Trace[913053617]: ---"Objects listed" error:Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90243": net/http: TLS handshake timeout 10001ms (06:42:11.177)
E0425 06:42:11.177211       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&resourceVersion=90243": net/http: TLS handshake timeout

Pod: kube-scheduler-k8s-kubeflow-master01 (Namespace: kube-system)

Describe (filtered)

      Reason:       Error

Logs (filtered)

W0425 06:43:37.338915       1 authentication.go:370] To require authentication configuration lookup to succeed, set --authentication-tolerate-lookup-failure=false
W0425 06:43:37.347849       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PodDisruptionBudget: Get "https://192.168.2.231:6443/apis/policy/v1/poddisruptionbudgets?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:37.347864       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Pod: Get "https://192.168.2.231:6443/api/v1/pods?fieldSelector=status.phase%21%3DSucceeded%2Cstatus.phase%21%3DFailed&limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:37.347931       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PodDisruptionBudget: failed to list *v1.PodDisruptionBudget: Get "https://192.168.2.231:6443/apis/policy/v1/poddisruptionbudgets?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:37.347931       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Pod: failed to list *v1.Pod: Get "https://192.168.2.231:6443/api/v1/pods?fieldSelector=status.phase%21%3DSucceeded%2Cstatus.phase%21%3DFailed&limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:37.347964       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:37.348054       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:37.348049       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.ReplicationController: Get "https://192.168.2.231:6443/api/v1/replicationcontrollers?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:37.348073       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.CSIStorageCapacity: Get "https://192.168.2.231:6443/apis/storage.k8s.io/v1/csistoragecapacities?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:37.348127       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.CSIStorageCapacity: failed to list *v1.CSIStorageCapacity: Get "https://192.168.2.231:6443/apis/storage.k8s.io/v1/csistoragecapacities?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:37.348096       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.ReplicationController: failed to list *v1.ReplicationController: Get "https://192.168.2.231:6443/api/v1/replicationcontrollers?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:37.348169       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.StorageClass: Get "https://192.168.2.231:6443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:37.348192       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PersistentVolume: Get "https://192.168.2.231:6443/api/v1/persistentvolumes?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:37.348248       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: Get "https://192.168.2.231:6443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:37.348255       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PersistentVolume: failed to list *v1.PersistentVolume: Get "https://192.168.2.231:6443/api/v1/persistentvolumes?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:37.348276       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.CSINode: Get "https://192.168.2.231:6443/apis/storage.k8s.io/v1/csinodes?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:37.348332       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.CSINode: failed to list *v1.CSINode: Get "https://192.168.2.231:6443/apis/storage.k8s.io/v1/csinodes?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:37.348349       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PersistentVolumeClaim: Get "https://192.168.2.231:6443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:37.348363       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.StatefulSet: Get "https://192.168.2.231:6443/apis/apps/v1/statefulsets?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:37.348416       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PersistentVolumeClaim: failed to list *v1.PersistentVolumeClaim: Get "https://192.168.2.231:6443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:37.348430       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: Get "https://192.168.2.231:6443/apis/apps/v1/statefulsets?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:37.348453       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:37.348500       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:37.348521       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Namespace: Get "https://192.168.2.231:6443/api/v1/namespaces?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:37.348563       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Namespace: failed to list *v1.Namespace: Get "https://192.168.2.231:6443/api/v1/namespaces?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:37.348627       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Get "https://192.168.2.231:6443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:37.348652       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.ReplicaSet: Get "https://192.168.2.231:6443/apis/apps/v1/replicasets?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:37.348693       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: Get "https://192.168.2.231:6443/apis/apps/v1/replicasets?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:37.348672       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https://192.168.2.231:6443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:37.348816       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.CSIDriver: Get "https://192.168.2.231:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:37.348863       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: Get "https://192.168.2.231:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:38.244059       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.CSIStorageCapacity: Get "https://192.168.2.231:6443/apis/storage.k8s.io/v1/csistoragecapacities?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:38.244111       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.CSIStorageCapacity: failed to list *v1.CSIStorageCapacity: Get "https://192.168.2.231:6443/apis/storage.k8s.io/v1/csistoragecapacities?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:38.266796       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Get "https://192.168.2.231:6443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:38.266837       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https://192.168.2.231:6443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:38.311510       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.ReplicationController: Get "https://192.168.2.231:6443/api/v1/replicationcontrollers?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:38.311561       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.ReplicationController: failed to list *v1.ReplicationController: Get "https://192.168.2.231:6443/api/v1/replicationcontrollers?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:38.321844       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.StorageClass: Get "https://192.168.2.231:6443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:38.321878       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: Get "https://192.168.2.231:6443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:38.406235       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PersistentVolume: Get "https://192.168.2.231:6443/api/v1/persistentvolumes?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:38.406310       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PersistentVolume: failed to list *v1.PersistentVolume: Get "https://192.168.2.231:6443/api/v1/persistentvolumes?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:38.552297       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.StatefulSet: Get "https://192.168.2.231:6443/apis/apps/v1/statefulsets?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:38.552341       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: Get "https://192.168.2.231:6443/apis/apps/v1/statefulsets?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:38.570941       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PersistentVolumeClaim: Get "https://192.168.2.231:6443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:38.570991       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PersistentVolumeClaim: failed to list *v1.PersistentVolumeClaim: Get "https://192.168.2.231:6443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:38.595447       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.CSINode: Get "https://192.168.2.231:6443/apis/storage.k8s.io/v1/csinodes?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:38.595504       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.CSINode: failed to list *v1.CSINode: Get "https://192.168.2.231:6443/apis/storage.k8s.io/v1/csinodes?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:38.616971       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Pod: Get "https://192.168.2.231:6443/api/v1/pods?fieldSelector=status.phase%21%3DSucceeded%2Cstatus.phase%21%3DFailed&limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:38.617010       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Pod: failed to list *v1.Pod: Get "https://192.168.2.231:6443/api/v1/pods?fieldSelector=status.phase%21%3DSucceeded%2Cstatus.phase%21%3DFailed&limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:38.681766       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:38.681824       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:38.706444       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PodDisruptionBudget: Get "https://192.168.2.231:6443/apis/policy/v1/poddisruptionbudgets?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:38.706503       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PodDisruptionBudget: failed to list *v1.PodDisruptionBudget: Get "https://192.168.2.231:6443/apis/policy/v1/poddisruptionbudgets?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:38.740204       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.CSIDriver: Get "https://192.168.2.231:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:38.740282       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: Get "https://192.168.2.231:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:38.793895       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:38.793947       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:38.925558       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.ReplicaSet: Get "https://192.168.2.231:6443/apis/apps/v1/replicasets?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:38.925621       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: Get "https://192.168.2.231:6443/apis/apps/v1/replicasets?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:38.933414       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Namespace: Get "https://192.168.2.231:6443/api/v1/namespaces?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:38.933476       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Namespace: failed to list *v1.Namespace: Get "https://192.168.2.231:6443/api/v1/namespaces?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:39.888461       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.CSIStorageCapacity: Get "https://192.168.2.231:6443/apis/storage.k8s.io/v1/csistoragecapacities?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:39.888551       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.CSIStorageCapacity: failed to list *v1.CSIStorageCapacity: Get "https://192.168.2.231:6443/apis/storage.k8s.io/v1/csistoragecapacities?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:39.917768       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.ReplicationController: Get "https://192.168.2.231:6443/api/v1/replicationcontrollers?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:39.917846       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.ReplicationController: failed to list *v1.ReplicationController: Get "https://192.168.2.231:6443/api/v1/replicationcontrollers?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:40.072819       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: Get "https://192.168.2.231:6443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:40.072898       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https://192.168.2.231:6443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:40.232696       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PersistentVolumeClaim: Get "https://192.168.2.231:6443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:40.232753       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PersistentVolumeClaim: failed to list *v1.PersistentVolumeClaim: Get "https://192.168.2.231:6443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:40.369325       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Pod: Get "https://192.168.2.231:6443/api/v1/pods?fieldSelector=status.phase%21%3DSucceeded%2Cstatus.phase%21%3DFailed&limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:40.369424       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Pod: failed to list *v1.Pod: Get "https://192.168.2.231:6443/api/v1/pods?fieldSelector=status.phase%21%3DSucceeded%2Cstatus.phase%21%3DFailed&limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:40.388857       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.CSIDriver: Get "https://192.168.2.231:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:40.388919       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: Get "https://192.168.2.231:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:40.485631       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.CSINode: Get "https://192.168.2.231:6443/apis/storage.k8s.io/v1/csinodes?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:40.485692       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.CSINode: failed to list *v1.CSINode: Get "https://192.168.2.231:6443/apis/storage.k8s.io/v1/csinodes?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:40.490082       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PersistentVolume: Get "https://192.168.2.231:6443/api/v1/persistentvolumes?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:40.490148       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PersistentVolume: failed to list *v1.PersistentVolume: Get "https://192.168.2.231:6443/api/v1/persistentvolumes?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:40.875919       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:40.875987       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://192.168.2.231:6443/api/v1/nodes?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:40.920333       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.StorageClass: Get "https://192.168.2.231:6443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:40.920415       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: Get "https://192.168.2.231:6443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:41.043071       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:41.043136       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:41.423338       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PodDisruptionBudget: Get "https://192.168.2.231:6443/apis/policy/v1/poddisruptionbudgets?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:41.423438       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PodDisruptionBudget: failed to list *v1.PodDisruptionBudget: Get "https://192.168.2.231:6443/apis/policy/v1/poddisruptionbudgets?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:41.437746       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.StatefulSet: Get "https://192.168.2.231:6443/apis/apps/v1/statefulsets?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:41.437801       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: Get "https://192.168.2.231:6443/apis/apps/v1/statefulsets?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:41.916364       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Namespace: Get "https://192.168.2.231:6443/api/v1/namespaces?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:41.916453       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Namespace: failed to list *v1.Namespace: Get "https://192.168.2.231:6443/api/v1/namespaces?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:42.018204       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.ReplicaSet: Get "https://192.168.2.231:6443/apis/apps/v1/replicasets?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:42.018283       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: Get "https://192.168.2.231:6443/apis/apps/v1/replicasets?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:43.327157       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.CSIStorageCapacity: Get "https://192.168.2.231:6443/apis/storage.k8s.io/v1/csistoragecapacities?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:43.327264       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.CSIStorageCapacity: failed to list *v1.CSIStorageCapacity: Get "https://192.168.2.231:6443/apis/storage.k8s.io/v1/csistoragecapacities?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:43.702493       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Pod: Get "https://192.168.2.231:6443/api/v1/pods?fieldSelector=status.phase%21%3DSucceeded%2Cstatus.phase%21%3DFailed&limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:43.702562       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Pod: failed to list *v1.Pod: Get "https://192.168.2.231:6443/api/v1/pods?fieldSelector=status.phase%21%3DSucceeded%2Cstatus.phase%21%3DFailed&limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:44.205139       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PersistentVolume: Get "https://192.168.2.231:6443/api/v1/persistentvolumes?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:44.205208       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PersistentVolume: failed to list *v1.PersistentVolume: Get "https://192.168.2.231:6443/api/v1/persistentvolumes?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:44.472849       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.CSINode: Get "https://192.168.2.231:6443/apis/storage.k8s.io/v1/csinodes?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:44.472901       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.CSINode: failed to list *v1.CSINode: Get "https://192.168.2.231:6443/apis/storage.k8s.io/v1/csinodes?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:44.508354       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.ReplicationController: Get "https://192.168.2.231:6443/api/v1/replicationcontrollers?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:44.508429       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.ReplicationController: failed to list *v1.ReplicationController: Get "https://192.168.2.231:6443/api/v1/replicationcontrollers?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:44.557290       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:44.557341       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://192.168.2.231:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:45.491873       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PersistentVolumeClaim: Get "https://192.168.2.231:6443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:45.491926       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PersistentVolumeClaim: failed to list *v1.PersistentVolumeClaim: Get "https://192.168.2.231:6443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:45.720126       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Namespace: Get "https://192.168.2.231:6443/api/v1/namespaces?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
E0425 06:43:45.720178       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Namespace: failed to list *v1.Namespace: Get "https://192.168.2.231:6443/api/v1/namespaces?limit=500&resourceVersion=0": dial tcp 192.168.2.231:6443: connect: connection refused
W0425 06:43:47.448069       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
E0425 06:43:47.448115       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
W0425 06:43:47.448161       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.ConfigMap: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot list resource "configmaps" in API group "" in the namespace "kube-system"
E0425 06:43:47.448189       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot list resource "configmaps" in API group "" in the namespace "kube-system"
W0425 06:43:47.448285       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0425 06:43:47.448307       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
W0425 06:43:47.448370       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csidrivers" in API group "storage.k8s.io" at the cluster scope
E0425 06:43:47.448411       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csidrivers" in API group "storage.k8s.io" at the cluster scope
W0425 06:43:47.448371       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
W0425 06:43:47.448432       1 reflector.go:539] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0425 06:43:47.448462       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0425 06:43:47.448435       1 reflector.go:147] k8s.io/client-go@v0.0.0/tools/cache/reflector.go:229: Failed to watch *v1.PodDisruptionBudget: failed to list *v1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope

Pod: ml-pipeline-ui-artifact-55f4d57d68-sb9sf (Namespace: kubeflow-user-example-com)

Logs (filtered)

[HPM] Subscribed to http-proxy events:  [ 'error', 'close' ]
[HPM] Subscribed to http-proxy events:  [ 'error', 'close' ]
[HPM] Subscribed to http-proxy events:  [ 'error', 'close' ]
[HPM] Subscribed to http-proxy events:  [ 'error', 'close' ]
[HPM] Subscribed to http-proxy events:  [ 'proxyReq', 'error', 'close' ]
[HPM] Subscribed to http-proxy events:  [ 'proxyReq', 'error', 'close' ]
[HPM] Subscribed to http-proxy events:  [ 'proxyReq', 'error', 'close' ]
[HPM] Subscribed to http-proxy events:  [ 'proxyReq', 'error', 'close' ]

Pod: pod-log-to-github-27mfh-system-container-driver-1041639969 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T08:20:41.830Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-07T08:20:41.830Z" level=error msg="cannot save parameter /tmp/outputs/pod-spec-patch" argo=true error="open /tmp/outputs/pod-spec-patch: no such file or directory"

Pod: pod-log-to-github-27mfh-system-container-driver-3397762364 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"70"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","20a1fb31-4328-4248-b2c5-96f538d96482","--dag_execution_id","70","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-27mfh/2025/05/07/pod-log-to-github-27mfh-system-container-driver-3397762364"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"70"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","20a1fb31-4328-4248-b2c5-96f538d96482","--dag_execution_id","70","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-27mfh/2025/05/07/pod-log-to-github-27mfh-system-container-driver-3397762364"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str = \"main\"\n):\n    import os\n    import subprocess\n\n\n    report_path = report_file.path\n    print(\"Checking if report exists at:\", report_path)\n    if not os.path.exists(report_path):\n        raise FileNotFoundError(f\"{report_path} does not exist\")\n\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    subprocess.run([\"cp\", report_path, os.path.join(repo_dir, \"report.md\")], check=True)\n\n    # Git user config\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", \"Add report\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(\"Report committed and pushed to GitHub.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"70"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","20a1fb31-4328-4248-b2c5-96f538d96482","--dag_execution_id","70","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-27mfh/2025/05/07/pod-log-to-github-27mfh-system-container-driver-3397762364"}}}

Logs (filtered)

time="2025-05-07T08:20:50.918Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-07T08:20:50.918Z" level=error msg="cannot save parameter /tmp/outputs/pod-spec-patch" argo=true error="open /tmp/outputs/pod-spec-patch: no such file or directory"

Pod: pod-log-to-github-27mfh-system-dag-driver-1764293623 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T08:20:31.731Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-2lp4x-system-container-driver-2962277050 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    print(\\\"Checking if report exists at:\\\", report_file.path)\\n    if not os.path.exists(report_file.path):\\n        raise FileNotFoundError(f\\\"{report_file.path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"47"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","ea0f30be-c0ab-40be-a31d-b6768793aa19","--dag_execution_id","47","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    print(\\\"Checking if report exists at:\\\", report_file.path)\\n    if not os.path.exists(report_file.path):\\n        raise FileNotFoundError(f\\\"{report_file.path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-2lp4x/2025/05/07/pod-log-to-github-2lp4x-system-container-driver-2962277050"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    print(\\\"Checking if report exists at:\\\", report_file.path)\\n    if not os.path.exists(report_file.path):\\n        raise FileNotFoundError(f\\\"{report_file.path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"47"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","ea0f30be-c0ab-40be-a31d-b6768793aa19","--dag_execution_id","47","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    print(\\\"Checking if report exists at:\\\", report_file.path)\\n    if not os.path.exists(report_file.path):\\n        raise FileNotFoundError(f\\\"{report_file.path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-2lp4x/2025/05/07/pod-log-to-github-2lp4x-system-container-driver-2962277050"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str = \"main\"\n):\n    import os\n    import subprocess\n\n    print(\"Checking if report exists at:\", report_file.path)\n    if not os.path.exists(report_file.path):\n        raise FileNotFoundError(f\"{report_file.path} does not exist\")\n\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    subprocess.run([\"cp\", report_path, os.path.join(repo_dir, \"report.md\")], check=True)\n\n    # Git user config\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", \"Add report\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(\"Report committed and pushed to GitHub.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    print(\\\"Checking if report exists at:\\\", report_file.path)\\n    if not os.path.exists(report_file.path):\\n        raise FileNotFoundError(f\\\"{report_file.path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"47"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","ea0f30be-c0ab-40be-a31d-b6768793aa19","--dag_execution_id","47","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    print(\\\"Checking if report exists at:\\\", report_file.path)\\n    if not os.path.exists(report_file.path):\\n        raise FileNotFoundError(f\\\"{report_file.path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-2lp4x/2025/05/07/pod-log-to-github-2lp4x-system-container-driver-2962277050"}}}

Logs (filtered)

time="2025-05-07T07:43:51.924Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-2lp4x-system-container-driver-3944702915 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T07:43:20.725Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-2lp4x-system-container-impl-2132725893 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T07:43:36.922Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-07T07:43:37.860Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-2lp4x-system-container-impl-424932916 (Namespace: kubeflow-user-example-com)

Describe (filtered)

Status:       Failed
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"ea0f30be-c0ab-40be-a31d-b6768793aa19\",\"--execution_id\",\"49\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"2\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ea0f30be-c0ab-40be-a31d-b6768793aa19/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    print(\\\"Checking if report exists at:\\\", report_file.path)\\n    if not os.path.exists(report_file.path):\\n        raise FileNotFoundError(f\\\"{report_file.path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-2lp4x/2025/05/07/pod-log-to-github-2lp4x-system-container-impl-424932916"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"ea0f30be-c0ab-40be-a31d-b6768793aa19\",\"--execution_id\",\"49\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"2\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ea0f30be-c0ab-40be-a31d-b6768793aa19/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    print(\\\"Checking if report exists at:\\\", report_file.path)\\n    if not os.path.exists(report_file.path):\\n        raise FileNotFoundError(f\\\"{report_file.path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"ea0f30be-c0ab-40be-a31d-b6768793aa19\",\"--execution_id\",\"49\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"2\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ea0f30be-c0ab-40be-a31d-b6768793aa19/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    print(\\\"Checking if report exists at:\\\", report_file.path)\\n    if not os.path.exists(report_file.path):\\n        raise FileNotFoundError(f\\\"{report_file.path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-2lp4x/2025/05/07/pod-log-to-github-2lp4x-system-container-impl-424932916"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"ea0f30be-c0ab-40be-a31d-b6768793aa19\",\"--execution_id\",\"49\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"2\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ea0f30be-c0ab-40be-a31d-b6768793aa19/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    print(\\\"Checking if report exists at:\\\", report_file.path)\\n    if not os.path.exists(report_file.path):\\n        raise FileNotFoundError(f\\\"{report_file.path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"ea0f30be-c0ab-40be-a31d-b6768793aa19\",\"--execution_id\",\"49\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"2\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ea0f30be-c0ab-40be-a31d-b6768793aa19/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    print(\\\"Checking if report exists at:\\\", report_file.path)\\n    if not os.path.exists(report_file.path):\\n        raise FileNotFoundError(f\\\"{report_file.path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-2lp4x/2025/05/07/pod-log-to-github-2lp4x-system-container-impl-424932916"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"ea0f30be-c0ab-40be-a31d-b6768793aa19\",\"--execution_id\",\"49\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"2\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ea0f30be-c0ab-40be-a31d-b6768793aa19/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    print(\\\"Checking if report exists at:\\\", report_file.path)\\n    if not os.path.exists(report_file.path):\\n        raise FileNotFoundError(f\\\"{report_file.path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
              raise FileNotFoundError(f"{report_file.path} does not exist")
      Reason:       Error
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"ea0f30be-c0ab-40be-a31d-b6768793aa19\",\"--execution_id\",\"49\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"2\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ea0f30be-c0ab-40be-a31d-b6768793aa19/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    print(\\\"Checking if report exists at:\\\", report_file.path)\\n    if not os.path.exists(report_file.path):\\n        raise FileNotFoundError(f\\\"{report_file.path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-2lp4x/2025/05/07/pod-log-to-github-2lp4x-system-container-impl-424932916"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"ea0f30be-c0ab-40be-a31d-b6768793aa19\",\"--execution_id\",\"49\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"2\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ea0f30be-c0ab-40be-a31d-b6768793aa19/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    print(\\\"Checking if report exists at:\\\", report_file.path)\\n    if not os.path.exists(report_file.path):\\n        raise FileNotFoundError(f\\\"{report_file.path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}

Logs (filtered)

Traceback (most recent call last):
F0507 07:44:09.885323      23 main.go:49] failed to execute component: exit status 1
time="2025-05-07T07:44:10.119Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-07T07:44:11.056Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-2lp4x-system-dag-driver-1238066821 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T07:43:11.638Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-4g9rw-system-container-driver-2913006772 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T08:45:16.224Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-07T08:45:16.224Z" level=error msg="cannot save parameter /tmp/outputs/pod-spec-patch" argo=true error="open /tmp/outputs/pod-spec-patch: no such file or directory"

Pod: pod-log-to-github-4g9rw-system-container-driver-4039953061 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"76"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","7c6cb0e8-e66f-4238-a5bf-3216b4a31add","--dag_execution_id","76","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-4g9rw/2025/05/07/pod-log-to-github-4g9rw-system-container-driver-4039953061"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"76"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","7c6cb0e8-e66f-4238-a5bf-3216b4a31add","--dag_execution_id","76","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-4g9rw/2025/05/07/pod-log-to-github-4g9rw-system-container-driver-4039953061"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str = \"main\"\n):\n    import os\n    import subprocess\n\n\n    report_path = report_file.path\n    print(\"Checking if report exists at:\", report_path)\n    if not os.path.exists(report_path):\n        raise FileNotFoundError(f\"{report_path} does not exist\")\n\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    subprocess.run([\"cp\", report_path, os.path.join(repo_dir, \"report.md\")], check=True)\n\n    # Git user config\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", \"Add report\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(\"Report committed and pushed to GitHub.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"76"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","7c6cb0e8-e66f-4238-a5bf-3216b4a31add","--dag_execution_id","76","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-4g9rw/2025/05/07/pod-log-to-github-4g9rw-system-container-driver-4039953061"}}}

Logs (filtered)

time="2025-05-07T08:45:26.324Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-07T08:45:26.324Z" level=error msg="cannot save parameter /tmp/outputs/pod-spec-patch" argo=true error="open /tmp/outputs/pod-spec-patch: no such file or directory"

Pod: pod-log-to-github-4g9rw-system-dag-driver-2674859260 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T08:45:07.124Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-5sf2x-system-container-driver-2781465620 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"99"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","c829eb92-c21e-4698-b639-9049977e037c","--dag_execution_id","99","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-5sf2x/2025/05/08/pod-log-to-github-5sf2x-system-container-driver-2781465620"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"99"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","c829eb92-c21e-4698-b639-9049977e037c","--dag_execution_id","99","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-5sf2x/2025/05/08/pod-log-to-github-5sf2x-system-container-driver-2781465620"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str = \"main\"\n):\n    import os\n    import subprocess\n\n\n    report_path = report_file.path\n    print(\"Checking if report exists at:\", report_path)\n    if not os.path.exists(report_path):\n        raise FileNotFoundError(f\"{report_path} does not exist\")\n\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    subprocess.run([\"cp\", report_path, os.path.join(repo_dir, \"report.md\")], check=True)\n\n    # Git user config\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", \"Add report\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(\"Report committed and pushed to GitHub.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"99"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","c829eb92-c21e-4698-b639-9049977e037c","--dag_execution_id","99","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-5sf2x/2025/05/08/pod-log-to-github-5sf2x-system-container-driver-2781465620"}}}

Logs (filtered)

time="2025-05-08T01:51:32.120Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-5sf2x-system-container-driver-4249229351 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T01:50:13.321Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-5sf2x-system-container-impl-3759230217 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T01:51:18.633Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T01:51:19.556Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-5sf2x-system-dag-driver-3768259711 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T01:50:03.123Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-6lxcf-system-container-driver-3868635348 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T07:34:18.413Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-6lxcf-system-container-impl-508915066 (Namespace: kubeflow-user-example-com)

Describe (filtered)

Status:       Failed
      Reason:       Error

Logs (filtered)

F0507 07:34:28.658339      24 main.go:49] failed to execute component: unable to create directory "/minio/mlpipeline/v2/artifacts/pod-log-to-github/0bf2c1c4-6741-4d39-a54c-24e49aa2d2ac/collect-pod-logs" for output artifact "output_report": mkdir /minio: permission denied
time="2025-05-07T07:34:29.467Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-07T07:34:30.407Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-6lxcf-system-dag-driver-709820316 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T07:34:08.223Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-7xk8h-system-container-driver-2972826129 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T06:54:26.622Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-7xk8h-system-container-impl-2399659743 (Namespace: kubeflow-user-example-com)

Describe (filtered)

Status:       Failed
      Reason:       Error

Logs (filtered)

F0507 06:54:36.932466      23 main.go:49] failed to execute component: unable to create directory "/minio/mlpipeline/v2/artifacts/pod-log-to-github/bc218fad-0847-42c8-a1fc-34af6cdff98d/collect-pod-logs" for output artifact "output_report": mkdir /minio: permission denied
time="2025-05-07T06:54:37.749Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-07T06:54:38.678Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-7xk8h-system-dag-driver-1001809255 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T06:54:16.530Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-85dff-system-container-driver-501119730 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"108"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","885ec7ea-92ea-45cc-8d87-9f6897397f67","--dag_execution_id","108","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-85dff/2025/05/08/pod-log-to-github-85dff-system-container-driver-501119730"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"108"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","885ec7ea-92ea-45cc-8d87-9f6897397f67","--dag_execution_id","108","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-85dff/2025/05/08/pod-log-to-github-85dff-system-container-driver-501119730"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","collect_filtered_logs"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \"crash\"):\n    import shutil\n    import json\n\n    tmp_report_path = \"/tmp/report.html\"\n    seen_namespaces = set()\n    namespace_events = {}\n\n    # kubectl 설치 방법 변경\n    subprocess.run([\"apt-get\", \"update\", \"-y\"], check=True)\n    subprocess.run([ \n        \"curl\", \"-LO\", \"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\"\n    ], check=True)\n    subprocess.run([\"chmod\", \"+x\", \"kubectl\"], check=True)\n    subprocess.run([\"mv\", \"kubectl\", \"/usr/local/bin/\"], check=True)\n\n    # Get all pods in all namespaces\n    pods_output = subprocess.getoutput(\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\")\n    pod_lines = pods_output.strip().split(\"\\n\")\n\n    with open(tmp_report_path, \"w\") as f:\n        f.write(f\"\u003chtml\u003e\u003cbody\u003e\u003ch1\u003e장애 복기 리포트 (keyword: `{keyword}`)\u003c/h1\u003e\\n\")\n\n        for line in pod_lines:\n            if not line.strip():\n                continue\n            try:\n                namespace, pod_name = line.split()\n            except ValueError:\n                continue\n\n            describe = subprocess.getoutput(f\"kubectl describe pod {pod_name} -n {namespace}\")\n            logs = subprocess.getoutput(f\"kubectl logs {pod_name} -n {namespace}\")\n\n            matched_describe = \"\\n\".join([l for l in describe.splitlines() if keyword in l])\n            matched_logs = \"\\n\".join([l for l in logs.splitlines() if keyword in l])\n\n            if matched_describe or matched_logs:\n                f.write(f\"\u003ch2\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\u003c/h2\u003e\\n\")\n                if matched_describe:\n                    f.write(f\"\u003ch3\u003eDescribe\u003c/h3\u003e\u003cpre\u003e{matched_describe}\u003c/pre\u003e\\n\")\n                if matched_logs:\n                    f.write(f\"\u003ch3\u003eLogs\u003c/h3\u003e\u003cpre\u003e{matched_logs}\u003c/pre\u003e\\n\")\n\n            if namespace not in seen_namespaces:\n                events = subprocess.getoutput(f\"kubectl get events -n {namespace}\")\n                matched_events = \"\\n\".join([l for l in events.splitlines() if keyword in l])\n                if matched_events:\n                    namespace_events[namespace] = matched_events\n                seen_namespaces.add(namespace)\n\n        # Write filtered events grouped by namespace\n        for ns, events in namespace_events.items():\n            f.write(f\"\u003ch3\u003eEvents in Namespace: `{ns}`\u003c/h3\u003e\u003cpre\u003e{events}\u003c/pre\u003e\\n\")\n\n        f.write(\"\u003c/body\u003e\u003c/html\u003e\")\n\n    subprocess.run([\"cp\", tmp_report_path, output_report.path], check=True)\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"108"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","885ec7ea-92ea-45cc-8d87-9f6897397f67","--dag_execution_id","108","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-85dff/2025/05/08/pod-log-to-github-85dff-system-container-driver-501119730"}}}

Logs (filtered)

time="2025-05-08T04:46:28.023Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-85dff-system-container-impl-1792978204 (Namespace: kubeflow-user-example-com)

Describe (filtered)

Status:       Failed
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"885ec7ea-92ea-45cc-8d87-9f6897397f67\",\"--execution_id\",\"109\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/885ec7ea-92ea-45cc-8d87-9f6897397f67/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-85dff/2025/05/08/pod-log-to-github-85dff-system-container-impl-1792978204"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"885ec7ea-92ea-45cc-8d87-9f6897397f67\",\"--execution_id\",\"109\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/885ec7ea-92ea-45cc-8d87-9f6897397f67/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"885ec7ea-92ea-45cc-8d87-9f6897397f67\",\"--execution_id\",\"109\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/885ec7ea-92ea-45cc-8d87-9f6897397f67/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-85dff/2025/05/08/pod-log-to-github-85dff-system-container-impl-1792978204"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"885ec7ea-92ea-45cc-8d87-9f6897397f67\",\"--execution_id\",\"109\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/885ec7ea-92ea-45cc-8d87-9f6897397f67/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"885ec7ea-92ea-45cc-8d87-9f6897397f67\",\"--execution_id\",\"109\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/885ec7ea-92ea-45cc-8d87-9f6897397f67/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-85dff/2025/05/08/pod-log-to-github-85dff-system-container-impl-1792978204"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"885ec7ea-92ea-45cc-8d87-9f6897397f67\",\"--execution_id\",\"109\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/885ec7ea-92ea-45cc-8d87-9f6897397f67/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
                  except ValueError:
      Reason:       Error
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"885ec7ea-92ea-45cc-8d87-9f6897397f67\",\"--execution_id\",\"109\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/885ec7ea-92ea-45cc-8d87-9f6897397f67/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-85dff/2025/05/08/pod-log-to-github-85dff-system-container-impl-1792978204"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"885ec7ea-92ea-45cc-8d87-9f6897397f67\",\"--execution_id\",\"109\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/885ec7ea-92ea-45cc-8d87-9f6897397f67/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}

Logs (filtered)

Traceback (most recent call last):
F0508 04:46:43.387390      24 main.go:49] failed to execute component: exit status 1
time="2025-05-08T04:46:44.231Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T04:46:45.171Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-85dff-system-dag-driver-2448589302 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T04:46:17.930Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-8mjdq-system-container-driver-3188370735 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"122"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","08b68c9b-5d11-4324-950f-8266d4d828e3","--dag_execution_id","122","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-8mjdq/2025/05/08/pod-log-to-github-8mjdq-system-container-driver-3188370735"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"122"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","08b68c9b-5d11-4324-950f-8266d4d828e3","--dag_execution_id","122","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-8mjdq/2025/05/08/pod-log-to-github-8mjdq-system-container-driver-3188370735"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","collect_filtered_logs"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \"crash\"):\n    import shutil\n    import json\n    import subprocess\n    import os\n\n    tmp_report_path = \"/tmp/report.html\"\n    seen_namespaces = set()\n    namespace_events = {}\n\n    # kubectl 설치 방법 변경\n    subprocess.run([\"apt-get\", \"update\", \"-y\"], check=True)\n    subprocess.run([ \n        \"curl\", \"-LO\", \"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\"\n    ], check=True)\n    subprocess.run([\"chmod\", \"+x\", \"kubectl\"], check=True)\n    subprocess.run([\"mv\", \"kubectl\", \"/usr/local/bin/\"], check=True)\n\n    # Get all pods in all namespaces\n    pods_output = subprocess.getoutput(\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\")\n    pod_lines = pods_output.strip().split(\"\\n\")\n\n    with open(tmp_report_path, \"w\") as f:\n        f.write(f\"\u003chtml\u003e\u003cbody\u003e\u003ch1\u003e장애 복기 리포트 (keyword: `{keyword}`)\u003c/h1\u003e\\n\")\n\n        for line in pod_lines:\n            if not line.strip():\n                continue\n            try:\n                namespace, pod_name = line.split()\n            except ValueError:\n                continue\n\n            describe = subprocess.getoutput(f\"kubectl describe pod {pod_name} -n {namespace}\")\n            logs = subprocess.getoutput(f\"kubectl logs {pod_name} -n {namespace}\")\n\n            matched_describe = \"\\n\".join([l for l in describe.splitlines() if keyword in l])\n            matched_logs = \"\\n\".join([l for l in logs.splitlines() if keyword in l])\n\n            if matched_describe or matched_logs:\n                f.write(f\"\u003ch2\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\u003c/h2\u003e\\n\")\n                if matched_describe:\n                    f.write(f\"\u003ch3\u003eDescribe\u003c/h3\u003e\u003cpre\u003e{matched_describe}\u003c/pre\u003e\\n\")\n                if matched_logs:\n                    f.write(f\"\u003ch3\u003eLogs\u003c/h3\u003e\u003cpre\u003e{matched_logs}\u003c/pre\u003e\\n\")\n\n            if namespace not in seen_namespaces:\n                events = subprocess.getoutput(f\"kubectl get events -n {namespace}\")\n                matched_events = \"\\n\".join([l for l in events.splitlines() if keyword in l])\n                if matched_events:\n                    namespace_events[namespace] = matched_events\n                seen_namespaces.add(namespace)\n\n        # Write filtered events grouped by namespace\n        for ns, events in namespace_events.items():\n            f.write(f\"\u003ch3\u003eEvents in Namespace: `{ns}`\u003c/h3\u003e\u003cpre\u003e{events}\u003c/pre\u003e\\n\")\n\n        f.write(\"\u003c/body\u003e\u003c/html\u003e\")\n\n    subprocess.run([\"cp\", tmp_report_path, output_report.path], check=True)\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"122"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","08b68c9b-5d11-4324-950f-8266d4d828e3","--dag_execution_id","122","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-8mjdq/2025/05/08/pod-log-to-github-8mjdq-system-container-driver-3188370735"}}}

Logs (filtered)

time="2025-05-08T05:26:05.127Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-8mjdq-system-container-driver-4088445008 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"122"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","08b68c9b-5d11-4324-950f-8266d4d828e3","--dag_execution_id","122","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-8mjdq/2025/05/08/pod-log-to-github-8mjdq-system-container-driver-4088445008"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"122"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","08b68c9b-5d11-4324-950f-8266d4d828e3","--dag_execution_id","122","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-8mjdq/2025/05/08/pod-log-to-github-8mjdq-system-container-driver-4088445008"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str = \"main\"\n):\n\n    import subprocess\n    import shutil\n    import os\n    from datetime import datetime\n\n    os.environ[\"GIT_ASKPASS\"] = \"echo\"\n\n    report_path = report_file.path\n    print(\"Checking if report exists at:\", report_path)\n    if not os.path.exists(report_path):\n        raise FileNotFoundError(f\"{report_path} does not exist\")\n\n    # HTML 파일 이름 만들기\n    timestamp = datetime.now().strftime(\"report-%Y%m%d-%H%M%S.html\")\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n    report_target_path = os.path.join(repo_dir, timestamp)\n\n    # Git clone\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    shutil.copyfile(report_path, report_target_path)\n\n    # Git config\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    # Git commit \u0026 push\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", f\"Add {timestamp}\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(f\"Report committed and pushed to GitHub as {timestamp}.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"122"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","08b68c9b-5d11-4324-950f-8266d4d828e3","--dag_execution_id","122","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-8mjdq/2025/05/08/pod-log-to-github-8mjdq-system-container-driver-4088445008"}}}

Logs (filtered)

time="2025-05-08T05:27:40.225Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-8mjdq-system-container-impl-2465145121 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"08b68c9b-5d11-4324-950f-8266d4d828e3\",\"--execution_id\",\"123\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/08b68c9b-5d11-4324-950f-8266d4d828e3/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-8mjdq/2025/05/08/pod-log-to-github-8mjdq-system-container-impl-2465145121"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"08b68c9b-5d11-4324-950f-8266d4d828e3\",\"--execution_id\",\"123\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/08b68c9b-5d11-4324-950f-8266d4d828e3/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"08b68c9b-5d11-4324-950f-8266d4d828e3\",\"--execution_id\",\"123\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/08b68c9b-5d11-4324-950f-8266d4d828e3/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-8mjdq/2025/05/08/pod-log-to-github-8mjdq-system-container-impl-2465145121"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"08b68c9b-5d11-4324-950f-8266d4d828e3\",\"--execution_id\",\"123\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/08b68c9b-5d11-4324-950f-8266d4d828e3/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"08b68c9b-5d11-4324-950f-8266d4d828e3\",\"--execution_id\",\"123\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/08b68c9b-5d11-4324-950f-8266d4d828e3/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-8mjdq/2025/05/08/pod-log-to-github-8mjdq-system-container-impl-2465145121"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"08b68c9b-5d11-4324-950f-8266d4d828e3\",\"--execution_id\",\"123\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/08b68c9b-5d11-4324-950f-8266d4d828e3/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
                  except ValueError:
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"08b68c9b-5d11-4324-950f-8266d4d828e3\",\"--execution_id\",\"123\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/08b68c9b-5d11-4324-950f-8266d4d828e3/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-8mjdq/2025/05/08/pod-log-to-github-8mjdq-system-container-impl-2465145121"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"08b68c9b-5d11-4324-950f-8266d4d828e3\",\"--execution_id\",\"123\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/08b68c9b-5d11-4324-950f-8266d4d828e3/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}

Logs (filtered)

time="2025-05-08T05:27:26.410Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T05:27:27.348Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-8mjdq-system-container-impl-3832912982 (Namespace: kubeflow-user-example-com)

Describe (filtered)

Status:       Failed
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"08b68c9b-5d11-4324-950f-8266d4d828e3\",\"--execution_id\",\"124\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"18\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/08b68c9b-5d11-4324-950f-8266d4d828e3/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-8mjdq/2025/05/08/pod-log-to-github-8mjdq-system-container-impl-3832912982"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"08b68c9b-5d11-4324-950f-8266d4d828e3\",\"--execution_id\",\"124\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"18\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/08b68c9b-5d11-4324-950f-8266d4d828e3/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"08b68c9b-5d11-4324-950f-8266d4d828e3\",\"--execution_id\",\"124\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"18\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/08b68c9b-5d11-4324-950f-8266d4d828e3/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-8mjdq/2025/05/08/pod-log-to-github-8mjdq-system-container-impl-3832912982"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"08b68c9b-5d11-4324-950f-8266d4d828e3\",\"--execution_id\",\"124\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"18\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/08b68c9b-5d11-4324-950f-8266d4d828e3/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"08b68c9b-5d11-4324-950f-8266d4d828e3\",\"--execution_id\",\"124\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"18\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/08b68c9b-5d11-4324-950f-8266d4d828e3/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-8mjdq/2025/05/08/pod-log-to-github-8mjdq-system-container-impl-3832912982"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"08b68c9b-5d11-4324-950f-8266d4d828e3\",\"--execution_id\",\"124\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"18\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/08b68c9b-5d11-4324-950f-8266d4d828e3/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
              raise FileNotFoundError(f"{report_path} does not exist")
      Reason:       Error
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"08b68c9b-5d11-4324-950f-8266d4d828e3\",\"--execution_id\",\"124\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"18\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/08b68c9b-5d11-4324-950f-8266d4d828e3/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-8mjdq/2025/05/08/pod-log-to-github-8mjdq-system-container-impl-3832912982"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"08b68c9b-5d11-4324-950f-8266d4d828e3\",\"--execution_id\",\"124\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"18\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/08b68c9b-5d11-4324-950f-8266d4d828e3/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}

Logs (filtered)

fatal: Authentication failed for 'https://github.com/Rosa1026/Kubeflow.git/'
Traceback (most recent call last):
F0508 05:27:59.501120      23 main.go:49] failed to execute component: exit status 1
time="2025-05-08T05:28:00.398Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T05:28:01.337Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-8mjdq-system-dag-driver-2380633379 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T05:25:55.017Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-cdd4s-system-container-driver-1975790569 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T07:49:57.328Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-07T07:49:57.328Z" level=error msg="cannot save parameter /tmp/outputs/pod-spec-patch" argo=true error="open /tmp/outputs/pod-spec-patch: no such file or directory"

Pod: pod-log-to-github-cdd4s-system-container-driver-2112860468 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    print(\\\"Checking if report exists at:\\\", report_file.path)\\n    if not os.path.exists(report_file.path):\\n        raise FileNotFoundError(f\\\"{report_file.path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_file, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"50"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","bb057cfa-9761-494a-a5bf-12ad12223771","--dag_execution_id","50","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    print(\\\"Checking if report exists at:\\\", report_file.path)\\n    if not os.path.exists(report_file.path):\\n        raise FileNotFoundError(f\\\"{report_file.path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_file, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-cdd4s/2025/05/07/pod-log-to-github-cdd4s-system-container-driver-2112860468"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    print(\\\"Checking if report exists at:\\\", report_file.path)\\n    if not os.path.exists(report_file.path):\\n        raise FileNotFoundError(f\\\"{report_file.path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_file, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"50"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","bb057cfa-9761-494a-a5bf-12ad12223771","--dag_execution_id","50","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    print(\\\"Checking if report exists at:\\\", report_file.path)\\n    if not os.path.exists(report_file.path):\\n        raise FileNotFoundError(f\\\"{report_file.path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_file, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-cdd4s/2025/05/07/pod-log-to-github-cdd4s-system-container-driver-2112860468"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str = \"main\"\n):\n    import os\n    import subprocess\n\n    print(\"Checking if report exists at:\", report_file.path)\n    if not os.path.exists(report_file.path):\n        raise FileNotFoundError(f\"{report_file.path} does not exist\")\n\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    subprocess.run([\"cp\", report_file, os.path.join(repo_dir, \"report.md\")], check=True)\n\n    # Git user config\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", \"Add report\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(\"Report committed and pushed to GitHub.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    print(\\\"Checking if report exists at:\\\", report_file.path)\\n    if not os.path.exists(report_file.path):\\n        raise FileNotFoundError(f\\\"{report_file.path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_file, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"50"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","bb057cfa-9761-494a-a5bf-12ad12223771","--dag_execution_id","50","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    print(\\\"Checking if report exists at:\\\", report_file.path)\\n    if not os.path.exists(report_file.path):\\n        raise FileNotFoundError(f\\\"{report_file.path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_file, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-cdd4s/2025/05/07/pod-log-to-github-cdd4s-system-container-driver-2112860468"}}}

Logs (filtered)

time="2025-05-07T07:50:07.429Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-cdd4s-system-container-impl-4284018394 (Namespace: kubeflow-user-example-com)

Describe (filtered)

Status:       Failed
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"bb057cfa-9761-494a-a5bf-12ad12223771\",\"--execution_id\",\"52\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"2\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ea0f30be-c0ab-40be-a31d-b6768793aa19/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    print(\\\"Checking if report exists at:\\\", report_file.path)\\n    if not os.path.exists(report_file.path):\\n        raise FileNotFoundError(f\\\"{report_file.path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_file, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-cdd4s/2025/05/07/pod-log-to-github-cdd4s-system-container-impl-4284018394"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"bb057cfa-9761-494a-a5bf-12ad12223771\",\"--execution_id\",\"52\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"2\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ea0f30be-c0ab-40be-a31d-b6768793aa19/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    print(\\\"Checking if report exists at:\\\", report_file.path)\\n    if not os.path.exists(report_file.path):\\n        raise FileNotFoundError(f\\\"{report_file.path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_file, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"bb057cfa-9761-494a-a5bf-12ad12223771\",\"--execution_id\",\"52\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"2\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ea0f30be-c0ab-40be-a31d-b6768793aa19/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    print(\\\"Checking if report exists at:\\\", report_file.path)\\n    if not os.path.exists(report_file.path):\\n        raise FileNotFoundError(f\\\"{report_file.path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_file, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-cdd4s/2025/05/07/pod-log-to-github-cdd4s-system-container-impl-4284018394"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"bb057cfa-9761-494a-a5bf-12ad12223771\",\"--execution_id\",\"52\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"2\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ea0f30be-c0ab-40be-a31d-b6768793aa19/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    print(\\\"Checking if report exists at:\\\", report_file.path)\\n    if not os.path.exists(report_file.path):\\n        raise FileNotFoundError(f\\\"{report_file.path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_file, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"bb057cfa-9761-494a-a5bf-12ad12223771\",\"--execution_id\",\"52\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"2\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ea0f30be-c0ab-40be-a31d-b6768793aa19/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    print(\\\"Checking if report exists at:\\\", report_file.path)\\n    if not os.path.exists(report_file.path):\\n        raise FileNotFoundError(f\\\"{report_file.path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_file, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-cdd4s/2025/05/07/pod-log-to-github-cdd4s-system-container-impl-4284018394"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"bb057cfa-9761-494a-a5bf-12ad12223771\",\"--execution_id\",\"52\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"2\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ea0f30be-c0ab-40be-a31d-b6768793aa19/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    print(\\\"Checking if report exists at:\\\", report_file.path)\\n    if not os.path.exists(report_file.path):\\n        raise FileNotFoundError(f\\\"{report_file.path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_file, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
              raise FileNotFoundError(f"{report_file.path} does not exist")
      Reason:       Error
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"bb057cfa-9761-494a-a5bf-12ad12223771\",\"--execution_id\",\"52\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"2\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ea0f30be-c0ab-40be-a31d-b6768793aa19/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    print(\\\"Checking if report exists at:\\\", report_file.path)\\n    if not os.path.exists(report_file.path):\\n        raise FileNotFoundError(f\\\"{report_file.path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_file, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-cdd4s/2025/05/07/pod-log-to-github-cdd4s-system-container-impl-4284018394"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"bb057cfa-9761-494a-a5bf-12ad12223771\",\"--execution_id\",\"52\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"2\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ea0f30be-c0ab-40be-a31d-b6768793aa19/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    print(\\\"Checking if report exists at:\\\", report_file.path)\\n    if not os.path.exists(report_file.path):\\n        raise FileNotFoundError(f\\\"{report_file.path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_file, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}

Logs (filtered)

Traceback (most recent call last):
F0507 07:50:25.216652      23 main.go:49] failed to execute component: exit status 1
time="2025-05-07T07:50:25.552Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-07T07:50:26.493Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-cdd4s-system-dag-driver-69468575 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T07:49:47.227Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-d289f-system-container-driver-3503402586 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T08:52:12.624Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-d289f-system-container-driver-753728567 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"79"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","6db52ab4-0016-4f69-9a09-ae7760c4e495","--dag_execution_id","79","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-d289f/2025/05/07/pod-log-to-github-d289f-system-container-driver-753728567"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"79"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","6db52ab4-0016-4f69-9a09-ae7760c4e495","--dag_execution_id","79","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-d289f/2025/05/07/pod-log-to-github-d289f-system-container-driver-753728567"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str = \"main\"\n):\n    import os\n    import subprocess\n\n\n    report_path = report_file.path\n    print(\"Checking if report exists at:\", report_path)\n    if not os.path.exists(report_path):\n        raise FileNotFoundError(f\"{report_path} does not exist\")\n\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    subprocess.run([\"cp\", report_path, os.path.join(repo_dir, \"report.md\")], check=True)\n\n    # Git user config\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", \"Add report\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(\"Report committed and pushed to GitHub.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"79"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","6db52ab4-0016-4f69-9a09-ae7760c4e495","--dag_execution_id","79","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-d289f/2025/05/07/pod-log-to-github-d289f-system-container-driver-753728567"}}}

Logs (filtered)

time="2025-05-07T08:52:52.114Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-d289f-system-container-impl-1617666516 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T08:52:37.889Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-07T08:52:38.824Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-d289f-system-container-impl-3303748921 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"6db52ab4-0016-4f69-9a09-ae7760c4e495\",\"--execution_id\",\"81\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"5\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/6db52ab4-0016-4f69-9a09-ae7760c4e495/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-d289f/2025/05/07/pod-log-to-github-d289f-system-container-impl-3303748921"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"6db52ab4-0016-4f69-9a09-ae7760c4e495\",\"--execution_id\",\"81\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"5\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/6db52ab4-0016-4f69-9a09-ae7760c4e495/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"6db52ab4-0016-4f69-9a09-ae7760c4e495\",\"--execution_id\",\"81\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"5\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/6db52ab4-0016-4f69-9a09-ae7760c4e495/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-d289f/2025/05/07/pod-log-to-github-d289f-system-container-impl-3303748921"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"6db52ab4-0016-4f69-9a09-ae7760c4e495\",\"--execution_id\",\"81\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"5\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/6db52ab4-0016-4f69-9a09-ae7760c4e495/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"6db52ab4-0016-4f69-9a09-ae7760c4e495\",\"--execution_id\",\"81\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"5\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/6db52ab4-0016-4f69-9a09-ae7760c4e495/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-d289f/2025/05/07/pod-log-to-github-d289f-system-container-impl-3303748921"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"6db52ab4-0016-4f69-9a09-ae7760c4e495\",\"--execution_id\",\"81\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"5\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/6db52ab4-0016-4f69-9a09-ae7760c4e495/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
              raise FileNotFoundError(f"{report_path} does not exist")
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"6db52ab4-0016-4f69-9a09-ae7760c4e495\",\"--execution_id\",\"81\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"5\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/6db52ab4-0016-4f69-9a09-ae7760c4e495/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-d289f/2025/05/07/pod-log-to-github-d289f-system-container-impl-3303748921"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"6db52ab4-0016-4f69-9a09-ae7760c4e495\",\"--execution_id\",\"81\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"5\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/6db52ab4-0016-4f69-9a09-ae7760c4e495/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}

Logs (filtered)

time="2025-05-07T08:53:12.334Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-07T08:53:13.278Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-d289f-system-dag-driver-87807846 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T08:52:02.527Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-djqnk-system-container-driver-1976872295 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T08:42:29.925Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-07T08:42:29.925Z" level=error msg="cannot save parameter /tmp/outputs/pod-spec-patch" argo=true error="open /tmp/outputs/pod-spec-patch: no such file or directory"

Pod: pod-log-to-github-djqnk-system-container-driver-4241663846 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"73"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","6c1650ed-1a6f-4e5e-8428-d8c4a1e14190","--dag_execution_id","73","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-djqnk/2025/05/07/pod-log-to-github-djqnk-system-container-driver-4241663846"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"73"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","6c1650ed-1a6f-4e5e-8428-d8c4a1e14190","--dag_execution_id","73","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-djqnk/2025/05/07/pod-log-to-github-djqnk-system-container-driver-4241663846"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str = \"main\"\n):\n    import os\n    import subprocess\n\n\n    report_path = report_file.path\n    print(\"Checking if report exists at:\", report_path)\n    if not os.path.exists(report_path):\n        raise FileNotFoundError(f\"{report_path} does not exist\")\n\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    subprocess.run([\"cp\", report_path, os.path.join(repo_dir, \"report.md\")], check=True)\n\n    # Git user config\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", \"Add report\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(\"Report committed and pushed to GitHub.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"73"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","6c1650ed-1a6f-4e5e-8428-d8c4a1e14190","--dag_execution_id","73","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-djqnk/2025/05/07/pod-log-to-github-djqnk-system-container-driver-4241663846"}}}

Logs (filtered)

time="2025-05-07T08:42:39.929Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-07T08:42:39.929Z" level=error msg="cannot save parameter /tmp/outputs/pod-spec-patch" argo=true error="open /tmp/outputs/pod-spec-patch: no such file or directory"

Pod: pod-log-to-github-djqnk-system-dag-driver-3911200761 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T08:42:19.816Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-dnplr-system-dag-driver-1081258443 (Namespace: kubeflow-user-example-com)

Describe (filtered)

Status:       Failed
      Reason:       Error
      Reason:       Error

Logs (filtered)

F0507 08:04:48.650837      15 main.go:79] KFP driver: driver.RootDAG(pipelineName=pod-log-to-github, runID=b8f973ef-b71b-4f91-9137-9209558f18ba, runtimeConfig, componentSpec) failed: configmaps "kfp-launcher" is forbidden: User "system:serviceaccount:kubeflow-user-example-com:default" cannot get resource "configmaps" in API group "" in the namespace "kubeflow-user-example-com"
time="2025-05-07T08:04:49.548Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-07T08:04:49.548Z" level=error msg="cannot save parameter /tmp/outputs/execution-id" argo=true error="open /tmp/outputs/execution-id: no such file or directory"
time="2025-05-07T08:04:49.548Z" level=error msg="cannot save parameter /tmp/outputs/iteration-count" argo=true error="open /tmp/outputs/iteration-count: no such file or directory"
time="2025-05-07T08:04:49.548Z" level=error msg="cannot save parameter /tmp/outputs/condition" argo=true error="open /tmp/outputs/condition: no such file or directory"

Pod: pod-log-to-github-gmf4s-system-container-driver-2119004180 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # 주요 키워드 목록\\n    describe_keywords = [\\\"CrashLoopBackOff\\\", \\\"OOMKilled\\\", \\\"Error\\\", \\\"Failed\\\", \\\"Back-off\\\"]\\n    log_keywords = [\\\"Traceback\\\", \\\"panic\\\", \\\"exception\\\", \\\"fail\\\", \\\"error\\\"]\\n    event_keywords = [\\\"Failed\\\", \\\"BackOff\\\", \\\"Unhealthy\\\", \\\"Killing\\\", \\\"Evicted\\\"]\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    def match_keywords(lines, keywords):\\n        return \\\"\\\\n\\\".join([line for line in lines if any(k in line for k in keywords)])\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        # HTML header\\n        f.write(\\\"\\\"\\\"\\n        \\u003chtml\\u003e\\n        \\u003chead\\u003e\\n            \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n            \\u003cstyle\\u003e\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\n                h1 { color: #2c3e50; }\\n                h2 { color: #34495e; }\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\n                section { margin-bottom: 30px; }\\n            \\u003c/style\\u003e\\n        \\u003c/head\\u003e\\n        \\u003cbody\\u003e\\n        \\u003ch1\\u003eKubernetes 장애 점검 리포트\\u003c/h1\\u003e\\n        \\\"\\\"\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe_output = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            log_output = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003ePod: {pod_name} (Namespace: {namespace})\\u003c/h2\\u003e\\\")\\n                if matched_describe:\\n                    f.write(\\\"\\u003ch3\\u003eDescribe (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_describe + \\\"\\u003c/pre\\u003e\\\")\\n                if matched_logs:\\n                    f.write(\\\"\\u003ch3\\u003eLogs (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_logs + \\\"\\u003c/pre\\u003e\\\")\\n                f.write(\\\"\\u003c/section\\u003e\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events_output = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003eEvents in Namespace: {ns}\\u003c/h2\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/section\\u003e\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"150"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","b9842c1b-438d-410c-ad18-f01e0145433b","--dag_execution_id","150","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # 주요 키워드 목록\\n    describe_keywords = [\\\"CrashLoopBackOff\\\", \\\"OOMKilled\\\", \\\"Error\\\", \\\"Failed\\\", \\\"Back-off\\\"]\\n    log_keywords = [\\\"Traceback\\\", \\\"panic\\\", \\\"exception\\\", \\\"fail\\\", \\\"error\\\"]\\n    event_keywords = [\\\"Failed\\\", \\\"BackOff\\\", \\\"Unhealthy\\\", \\\"Killing\\\", \\\"Evicted\\\"]\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    def match_keywords(lines, keywords):\\n        return \\\"\\\\n\\\".join([line for line in lines if any(k in line for k in keywords)])\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        # HTML header\\n        f.write(\\\"\\\"\\\"\\n        \\u003chtml\\u003e\\n        \\u003chead\\u003e\\n            \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n            \\u003cstyle\\u003e\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\n                h1 { color: #2c3e50; }\\n                h2 { color: #34495e; }\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\n                section { margin-bottom: 30px; }\\n            \\u003c/style\\u003e\\n        \\u003c/head\\u003e\\n        \\u003cbody\\u003e\\n        \\u003ch1\\u003eKubernetes 장애 점검 리포트\\u003c/h1\\u003e\\n        \\\"\\\"\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe_output = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            log_output = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003ePod: {pod_name} (Namespace: {namespace})\\u003c/h2\\u003e\\\")\\n                if matched_describe:\\n                    f.write(\\\"\\u003ch3\\u003eDescribe (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_describe + \\\"\\u003c/pre\\u003e\\\")\\n                if matched_logs:\\n                    f.write(\\\"\\u003ch3\\u003eLogs (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_logs + \\\"\\u003c/pre\\u003e\\\")\\n                f.write(\\\"\\u003c/section\\u003e\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events_output = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003eEvents in Namespace: {ns}\\u003c/h2\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/section\\u003e\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-gmf4s/2025/05/08/pod-log-to-github-gmf4s-system-container-driver-2119004180"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # 주요 키워드 목록\\n    describe_keywords = [\\\"CrashLoopBackOff\\\", \\\"OOMKilled\\\", \\\"Error\\\", \\\"Failed\\\", \\\"Back-off\\\"]\\n    log_keywords = [\\\"Traceback\\\", \\\"panic\\\", \\\"exception\\\", \\\"fail\\\", \\\"error\\\"]\\n    event_keywords = [\\\"Failed\\\", \\\"BackOff\\\", \\\"Unhealthy\\\", \\\"Killing\\\", \\\"Evicted\\\"]\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    def match_keywords(lines, keywords):\\n        return \\\"\\\\n\\\".join([line for line in lines if any(k in line for k in keywords)])\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        # HTML header\\n        f.write(\\\"\\\"\\\"\\n        \\u003chtml\\u003e\\n        \\u003chead\\u003e\\n            \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n            \\u003cstyle\\u003e\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\n                h1 { color: #2c3e50; }\\n                h2 { color: #34495e; }\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\n                section { margin-bottom: 30px; }\\n            \\u003c/style\\u003e\\n        \\u003c/head\\u003e\\n        \\u003cbody\\u003e\\n        \\u003ch1\\u003eKubernetes 장애 점검 리포트\\u003c/h1\\u003e\\n        \\\"\\\"\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe_output = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            log_output = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003ePod: {pod_name} (Namespace: {namespace})\\u003c/h2\\u003e\\\")\\n                if matched_describe:\\n                    f.write(\\\"\\u003ch3\\u003eDescribe (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_describe + \\\"\\u003c/pre\\u003e\\\")\\n                if matched_logs:\\n                    f.write(\\\"\\u003ch3\\u003eLogs (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_logs + \\\"\\u003c/pre\\u003e\\\")\\n                f.write(\\\"\\u003c/section\\u003e\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events_output = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003eEvents in Namespace: {ns}\\u003c/h2\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/section\\u003e\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"150"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","b9842c1b-438d-410c-ad18-f01e0145433b","--dag_execution_id","150","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # 주요 키워드 목록\\n    describe_keywords = [\\\"CrashLoopBackOff\\\", \\\"OOMKilled\\\", \\\"Error\\\", \\\"Failed\\\", \\\"Back-off\\\"]\\n    log_keywords = [\\\"Traceback\\\", \\\"panic\\\", \\\"exception\\\", \\\"fail\\\", \\\"error\\\"]\\n    event_keywords = [\\\"Failed\\\", \\\"BackOff\\\", \\\"Unhealthy\\\", \\\"Killing\\\", \\\"Evicted\\\"]\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    def match_keywords(lines, keywords):\\n        return \\\"\\\\n\\\".join([line for line in lines if any(k in line for k in keywords)])\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        # HTML header\\n        f.write(\\\"\\\"\\\"\\n        \\u003chtml\\u003e\\n        \\u003chead\\u003e\\n            \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n            \\u003cstyle\\u003e\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\n                h1 { color: #2c3e50; }\\n                h2 { color: #34495e; }\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\n                section { margin-bottom: 30px; }\\n            \\u003c/style\\u003e\\n        \\u003c/head\\u003e\\n        \\u003cbody\\u003e\\n        \\u003ch1\\u003eKubernetes 장애 점검 리포트\\u003c/h1\\u003e\\n        \\\"\\\"\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe_output = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            log_output = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003ePod: {pod_name} (Namespace: {namespace})\\u003c/h2\\u003e\\\")\\n                if matched_describe:\\n                    f.write(\\\"\\u003ch3\\u003eDescribe (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_describe + \\\"\\u003c/pre\\u003e\\\")\\n                if matched_logs:\\n                    f.write(\\\"\\u003ch3\\u003eLogs (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_logs + \\\"\\u003c/pre\\u003e\\\")\\n                f.write(\\\"\\u003c/section\\u003e\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events_output = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003eEvents in Namespace: {ns}\\u003c/h2\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/section\\u003e\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-gmf4s/2025/05/08/pod-log-to-github-gmf4s-system-container-driver-2119004180"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","collect_filtered_logs"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef collect_filtered_logs(output_report: Output[Dataset]):\n    import subprocess\n    import os\n\n    tmp_report_path = \"/tmp/report.html\"\n    seen_namespaces = set()\n    namespace_events = {}\n\n    # 주요 키워드 목록\n    describe_keywords = [\"CrashLoopBackOff\", \"OOMKilled\", \"Error\", \"Failed\", \"Back-off\"]\n    log_keywords = [\"Traceback\", \"panic\", \"exception\", \"fail\", \"error\"]\n    event_keywords = [\"Failed\", \"BackOff\", \"Unhealthy\", \"Killing\", \"Evicted\"]\n\n    # kubectl 설치\n    subprocess.run([\"apt-get\", \"update\", \"-y\"], check=True)\n    subprocess.run([\n        \"curl\", \"-LO\", \"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\"\n    ], check=True)\n    subprocess.run([\"chmod\", \"+x\", \"kubectl\"], check=True)\n    subprocess.run([\"mv\", \"kubectl\", \"/usr/local/bin/\"], check=True)\n\n    # Get all pods\n    pods_output = subprocess.getoutput(\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\")\n    pod_lines = pods_output.strip().split(\"\\n\")\n\n    def match_keywords(lines, keywords):\n        return \"\\n\".join([line for line in lines if any(k in line for k in keywords)])\n\n    with open(tmp_report_path, \"w\") as f:\n        # HTML header\n        f.write(\"\"\"\n        \u003chtml\u003e\n        \u003chead\u003e\n            \u003cmeta charset=\"UTF-8\"\u003e\n            \u003cstyle\u003e\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\n                h1 { color: #2c3e50; }\n                h2 { color: #34495e; }\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\n                section { margin-bottom: 30px; }\n            \u003c/style\u003e\n        \u003c/head\u003e\n        \u003cbody\u003e\n        \u003ch1\u003eKubernetes 장애 점검 리포트\u003c/h1\u003e\n        \"\"\")\n\n        for line in pod_lines:\n            if not line.strip():\n                continue\n            try:\n                namespace, pod_name = line.split()\n            except ValueError:\n                continue\n\n            describe_output = subprocess.getoutput(f\"kubectl describe pod {pod_name} -n {namespace}\")\n            log_output = subprocess.getoutput(f\"kubectl logs {pod_name} -n {namespace}\")\n\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\n\n            if matched_describe or matched_logs:\n                f.write(f\"\u003csection\u003e\u003ch2\u003ePod: {pod_name} (Namespace: {namespace})\u003c/h2\u003e\")\n                if matched_describe:\n                    f.write(\"\u003ch3\u003eDescribe (filtered)\u003c/h3\u003e\u003cpre\u003e\" + matched_describe + \"\u003c/pre\u003e\")\n                if matched_logs:\n                    f.write(\"\u003ch3\u003eLogs (filtered)\u003c/h3\u003e\u003cpre\u003e\" + matched_logs + \"\u003c/pre\u003e\")\n                f.write(\"\u003c/section\u003e\")\n\n            if namespace not in seen_namespaces:\n                events_output = subprocess.getoutput(f\"kubectl get events -n {namespace}\")\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\n                if matched_events:\n                    namespace_events[namespace] = matched_events\n                seen_namespaces.add(namespace)\n\n        for ns, events in namespace_events.items():\n            f.write(f\"\u003csection\u003e\u003ch2\u003eEvents in Namespace: {ns}\u003c/h2\u003e\u003cpre\u003e{events}\u003c/pre\u003e\u003c/section\u003e\")\n\n        f.write(\"\u003c/body\u003e\u003c/html\u003e\")\n\n    subprocess.run([\"cp\", tmp_report_path, output_report.path], check=True)\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # 주요 키워드 목록\\n    describe_keywords = [\\\"CrashLoopBackOff\\\", \\\"OOMKilled\\\", \\\"Error\\\", \\\"Failed\\\", \\\"Back-off\\\"]\\n    log_keywords = [\\\"Traceback\\\", \\\"panic\\\", \\\"exception\\\", \\\"fail\\\", \\\"error\\\"]\\n    event_keywords = [\\\"Failed\\\", \\\"BackOff\\\", \\\"Unhealthy\\\", \\\"Killing\\\", \\\"Evicted\\\"]\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    def match_keywords(lines, keywords):\\n        return \\\"\\\\n\\\".join([line for line in lines if any(k in line for k in keywords)])\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        # HTML header\\n        f.write(\\\"\\\"\\\"\\n        \\u003chtml\\u003e\\n        \\u003chead\\u003e\\n            \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n            \\u003cstyle\\u003e\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\n                h1 { color: #2c3e50; }\\n                h2 { color: #34495e; }\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\n                section { margin-bottom: 30px; }\\n            \\u003c/style\\u003e\\n        \\u003c/head\\u003e\\n        \\u003cbody\\u003e\\n        \\u003ch1\\u003eKubernetes 장애 점검 리포트\\u003c/h1\\u003e\\n        \\\"\\\"\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe_output = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            log_output = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003ePod: {pod_name} (Namespace: {namespace})\\u003c/h2\\u003e\\\")\\n                if matched_describe:\\n                    f.write(\\\"\\u003ch3\\u003eDescribe (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_describe + \\\"\\u003c/pre\\u003e\\\")\\n                if matched_logs:\\n                    f.write(\\\"\\u003ch3\\u003eLogs (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_logs + \\\"\\u003c/pre\\u003e\\\")\\n                f.write(\\\"\\u003c/section\\u003e\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events_output = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003eEvents in Namespace: {ns}\\u003c/h2\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/section\\u003e\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"150"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","b9842c1b-438d-410c-ad18-f01e0145433b","--dag_execution_id","150","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # 주요 키워드 목록\\n    describe_keywords = [\\\"CrashLoopBackOff\\\", \\\"OOMKilled\\\", \\\"Error\\\", \\\"Failed\\\", \\\"Back-off\\\"]\\n    log_keywords = [\\\"Traceback\\\", \\\"panic\\\", \\\"exception\\\", \\\"fail\\\", \\\"error\\\"]\\n    event_keywords = [\\\"Failed\\\", \\\"BackOff\\\", \\\"Unhealthy\\\", \\\"Killing\\\", \\\"Evicted\\\"]\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    def match_keywords(lines, keywords):\\n        return \\\"\\\\n\\\".join([line for line in lines if any(k in line for k in keywords)])\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        # HTML header\\n        f.write(\\\"\\\"\\\"\\n        \\u003chtml\\u003e\\n        \\u003chead\\u003e\\n            \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n            \\u003cstyle\\u003e\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\n                h1 { color: #2c3e50; }\\n                h2 { color: #34495e; }\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\n                section { margin-bottom: 30px; }\\n            \\u003c/style\\u003e\\n        \\u003c/head\\u003e\\n        \\u003cbody\\u003e\\n        \\u003ch1\\u003eKubernetes 장애 점검 리포트\\u003c/h1\\u003e\\n        \\\"\\\"\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe_output = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            log_output = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003ePod: {pod_name} (Namespace: {namespace})\\u003c/h2\\u003e\\\")\\n                if matched_describe:\\n                    f.write(\\\"\\u003ch3\\u003eDescribe (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_describe + \\\"\\u003c/pre\\u003e\\\")\\n                if matched_logs:\\n                    f.write(\\\"\\u003ch3\\u003eLogs (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_logs + \\\"\\u003c/pre\\u003e\\\")\\n                f.write(\\\"\\u003c/section\\u003e\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events_output = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003eEvents in Namespace: {ns}\\u003c/h2\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/section\\u003e\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-gmf4s/2025/05/08/pod-log-to-github-gmf4s-system-container-driver-2119004180"}}}

Logs (filtered)

    "\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef collect_filtered_logs(output_report: Output[Dataset]):\n    import subprocess\n    import os\n\n    tmp_report_path = \"/tmp/report.html\"\n    seen_namespaces = set()\n    namespace_events = {}\n\n    # 주요 키워드 목록\n    describe_keywords = [\"CrashLoopBackOff\", \"OOMKilled\", \"Error\", \"Failed\", \"Back-off\"]\n    log_keywords = [\"Traceback\", \"panic\", \"exception\", \"fail\", \"error\"]\n    event_keywords = [\"Failed\", \"BackOff\", \"Unhealthy\", \"Killing\", \"Evicted\"]\n\n    # kubectl 설치\n    subprocess.run([\"apt-get\", \"update\", \"-y\"], check=True)\n    subprocess.run([\n        \"curl\", \"-LO\", \"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\"\n    ], check=True)\n    subprocess.run([\"chmod\", \"+x\", \"kubectl\"], check=True)\n    subprocess.run([\"mv\", \"kubectl\", \"/usr/local/bin/\"], check=True)\n\n    # Get all pods\n    pods_output = subprocess.getoutput(\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\")\n    pod_lines = pods_output.strip().split(\"\\n\")\n\n    def match_keywords(lines, keywords):\n        return \"\\n\".join([line for line in lines if any(k in line for k in keywords)])\n\n    with open(tmp_report_path, \"w\") as f:\n        # HTML header\n        f.write(\"\"\"\n        \u003chtml\u003e\n        \u003chead\u003e\n            \u003cmeta charset=\"UTF-8\"\u003e\n            \u003cstyle\u003e\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\n                h1 { color: #2c3e50; }\n                h2 { color: #34495e; }\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\n                section { margin-bottom: 30px; }\n            \u003c/style\u003e\n        \u003c/head\u003e\n        \u003cbody\u003e\n        \u003ch1\u003eKubernetes 장애 점검 리포트\u003c/h1\u003e\n        \"\"\")\n\n        for line in pod_lines:\n            if not line.strip():\n                continue\n            try:\n                namespace, pod_name = line.split()\n            except ValueError:\n                continue\n\n            describe_output = subprocess.getoutput(f\"kubectl describe pod {pod_name} -n {namespace}\")\n            log_output = subprocess.getoutput(f\"kubectl logs {pod_name} -n {namespace}\")\n\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\n\n            if matched_describe or matched_logs:\n                f.write(f\"\u003csection\u003e\u003ch2\u003ePod: {pod_name} (Namespace: {namespace})\u003c/h2\u003e\")\n                if matched_describe:\n                    f.write(\"\u003ch3\u003eDescribe (filtered)\u003c/h3\u003e\u003cpre\u003e\" + matched_describe + \"\u003c/pre\u003e\")\n                if matched_logs:\n                    f.write(\"\u003ch3\u003eLogs (filtered)\u003c/h3\u003e\u003cpre\u003e\" + matched_logs + \"\u003c/pre\u003e\")\n                f.write(\"\u003c/section\u003e\")\n\n            if namespace not in seen_namespaces:\n                events_output = subprocess.getoutput(f\"kubectl get events -n {namespace}\")\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\n                if matched_events:\n                    namespace_events[namespace] = matched_events\n                seen_namespaces.add(namespace)\n\n        for ns, events in namespace_events.items():\n            f.write(f\"\u003csection\u003e\u003ch2\u003eEvents in Namespace: {ns}\u003c/h2\u003e\u003cpre\u003e{events}\u003c/pre\u003e\u003c/section\u003e\")\n\n        f.write(\"\u003c/body\u003e\u003c/html\u003e\")\n\n    subprocess.run([\"cp\", tmp_report_path, output_report.path], check=True)\n\n"
{"containers":[{"name":"main","image":"python:3.9","command":["/var/run/argo/argoexec","emissary","--","/kfp-launcher/launch","--pipeline_name","pod-log-to-github","--run_id","b9842c1b-438d-410c-ad18-f01e0145433b","--execution_id","151","--executor_input","{\"inputs\":{}, \"outputs\":{\"artifacts\":{\"output_report\":{\"artifacts\":[{\"type\":{\"schemaTitle\":\"system.Dataset\", \"schemaVersion\":\"0.0.1\"}, \"uri\":\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b9842c1b-438d-410c-ad18-f01e0145433b/collect-filtered-logs/output_report\"}]}}, \"outputFile\":\"/tmp/kfp_outputs/output_metadata.json\"}}","--component_spec","{\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\", \"schemaVersion\":\"0.0.1\"}}}}, \"executorLabel\":\"exec-collect-filtered-logs\"}","--pod_name","$(KFP_POD_NAME)","--pod_uid","$(KFP_POD_UID)","--mlmd_server_address","$(METADATA_GRPC_SERVICE_HOST)","--mlmd_server_port","$(METADATA_GRPC_SERVICE_PORT)","--"],"args":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef collect_filtered_logs(output_report: Output[Dataset]):\n    import subprocess\n    import os\n\n    tmp_report_path = \"/tmp/report.html\"\n    seen_namespaces = set()\n    namespace_events = {}\n\n    # 주요 키워드 목록\n    describe_keywords = [\"CrashLoopBackOff\", \"OOMKilled\", \"Error\", \"Failed\", \"Back-off\"]\n    log_keywords = [\"Traceback\", \"panic\", \"exception\", \"fail\", \"error\"]\n    event_keywords = [\"Failed\", \"BackOff\", \"Unhealthy\", \"Killing\", \"Evicted\"]\n\n    # kubectl 설치\n    subprocess.run([\"apt-get\", \"update\", \"-y\"], check=True)\n    subprocess.run([\n        \"curl\", \"-LO\", \"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\"\n    ], check=True)\n    subprocess.run([\"chmod\", \"+x\", \"kubectl\"], check=True)\n    subprocess.run([\"mv\", \"kubectl\", \"/usr/local/bin/\"], check=True)\n\n    # Get all pods\n    pods_output = subprocess.getoutput(\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\")\n    pod_lines = pods_output.strip().split(\"\\n\")\n\n    def match_keywords(lines, keywords):\n        return \"\\n\".join([line for line in lines if any(k in line for k in keywords)])\n\n    with open(tmp_report_path, \"w\") as f:\n        # HTML header\n        f.write(\"\"\"\n        \u003chtml\u003e\n        \u003chead\u003e\n            \u003cmeta charset=\"UTF-8\"\u003e\n            \u003cstyle\u003e\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\n                h1 { color: #2c3e50; }\n                h2 { color: #34495e; }\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\n                section { margin-bottom: 30px; }\n            \u003c/style\u003e\n        \u003c/head\u003e\n        \u003cbody\u003e\n        \u003ch1\u003eKubernetes 장애 점검 리포트\u003c/h1\u003e\n        \"\"\")\n\n        for line in pod_lines:\n            if not line.strip():\n                continue\n            try:\n                namespace, pod_name = line.split()\n            except ValueError:\n                continue\n\n            describe_output = subprocess.getoutput(f\"kubectl describe pod {pod_name} -n {namespace}\")\n            log_output = subprocess.getoutput(f\"kubectl logs {pod_name} -n {namespace}\")\n\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\n\n            if matched_describe or matched_logs:\n                f.write(f\"\u003csection\u003e\u003ch2\u003ePod: {pod_name} (Namespace: {namespace})\u003c/h2\u003e\")\n                if matched_describe:\n                    f.write(\"\u003ch3\u003eDescribe (filtered)\u003c/h3\u003e\u003cpre\u003e\" + matched_describe + \"\u003c/pre\u003e\")\n                if matched_logs:\n                    f.write(\"\u003ch3\u003eLogs (filtered)\u003c/h3\u003e\u003cpre\u003e\" + matched_logs + \"\u003c/pre\u003e\")\n                f.write(\"\u003c/section\u003e\")\n\n            if namespace not in seen_namespaces:\n                events_output = subprocess.getoutput(f\"kubectl get events -n {namespace}\")\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\n                if matched_events:\n                    namespace_events[namespace] = matched_events\n                seen_namespaces.add(namespace)\n\n        for ns, events in namespace_events.items():\n            f.write(f\"\u003csection\u003e\u003ch2\u003eEvents in Namespace: {ns}\u003c/h2\u003e\u003cpre\u003e{events}\u003c/pre\u003e\u003c/section\u003e\")\n\n        f.write(\"\u003c/body\u003e\u003c/html\u003e\")\n\n    subprocess.run([\"cp\", tmp_report_path, output_report.path], check=True)\n\n","--executor_input","{{$}}","--function_to_execute","collect_filtered_logs"],"resources":{}}]}
time="2025-05-08T06:35:57.416Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-gmf4s-system-container-impl-1948391354 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"--execution_id\",\"151\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b9842c1b-438d-410c-ad18-f01e0145433b/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # 주요 키워드 목록\\n    describe_keywords = [\\\"CrashLoopBackOff\\\", \\\"OOMKilled\\\", \\\"Error\\\", \\\"Failed\\\", \\\"Back-off\\\"]\\n    log_keywords = [\\\"Traceback\\\", \\\"panic\\\", \\\"exception\\\", \\\"fail\\\", \\\"error\\\"]\\n    event_keywords = [\\\"Failed\\\", \\\"BackOff\\\", \\\"Unhealthy\\\", \\\"Killing\\\", \\\"Evicted\\\"]\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    def match_keywords(lines, keywords):\\n        return \\\"\\\\n\\\".join([line for line in lines if any(k in line for k in keywords)])\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        # HTML header\\n        f.write(\\\"\\\"\\\"\\n        \\u003chtml\\u003e\\n        \\u003chead\\u003e\\n            \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n            \\u003cstyle\\u003e\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\n                h1 { color: #2c3e50; }\\n                h2 { color: #34495e; }\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\n                section { margin-bottom: 30px; }\\n            \\u003c/style\\u003e\\n        \\u003c/head\\u003e\\n        \\u003cbody\\u003e\\n        \\u003ch1\\u003eKubernetes 장애 점검 리포트\\u003c/h1\\u003e\\n        \\\"\\\"\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe_output = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            log_output = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003ePod: {pod_name} (Namespace: {namespace})\\u003c/h2\\u003e\\\")\\n                if matched_describe:\\n                    f.write(\\\"\\u003ch3\\u003eDescribe (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_describe + \\\"\\u003c/pre\\u003e\\\")\\n                if matched_logs:\\n                    f.write(\\\"\\u003ch3\\u003eLogs (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_logs + \\\"\\u003c/pre\\u003e\\\")\\n                f.write(\\\"\\u003c/section\\u003e\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events_output = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003eEvents in Namespace: {ns}\\u003c/h2\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/section\\u003e\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-gmf4s/2025/05/08/pod-log-to-github-gmf4s-system-container-impl-1948391354"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"--execution_id\",\"151\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b9842c1b-438d-410c-ad18-f01e0145433b/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # 주요 키워드 목록\\n    describe_keywords = [\\\"CrashLoopBackOff\\\", \\\"OOMKilled\\\", \\\"Error\\\", \\\"Failed\\\", \\\"Back-off\\\"]\\n    log_keywords = [\\\"Traceback\\\", \\\"panic\\\", \\\"exception\\\", \\\"fail\\\", \\\"error\\\"]\\n    event_keywords = [\\\"Failed\\\", \\\"BackOff\\\", \\\"Unhealthy\\\", \\\"Killing\\\", \\\"Evicted\\\"]\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    def match_keywords(lines, keywords):\\n        return \\\"\\\\n\\\".join([line for line in lines if any(k in line for k in keywords)])\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        # HTML header\\n        f.write(\\\"\\\"\\\"\\n        \\u003chtml\\u003e\\n        \\u003chead\\u003e\\n            \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n            \\u003cstyle\\u003e\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\n                h1 { color: #2c3e50; }\\n                h2 { color: #34495e; }\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\n                section { margin-bottom: 30px; }\\n            \\u003c/style\\u003e\\n        \\u003c/head\\u003e\\n        \\u003cbody\\u003e\\n        \\u003ch1\\u003eKubernetes 장애 점검 리포트\\u003c/h1\\u003e\\n        \\\"\\\"\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe_output = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            log_output = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003ePod: {pod_name} (Namespace: {namespace})\\u003c/h2\\u003e\\\")\\n                if matched_describe:\\n                    f.write(\\\"\\u003ch3\\u003eDescribe (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_describe + \\\"\\u003c/pre\\u003e\\\")\\n                if matched_logs:\\n                    f.write(\\\"\\u003ch3\\u003eLogs (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_logs + \\\"\\u003c/pre\\u003e\\\")\\n                f.write(\\\"\\u003c/section\\u003e\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events_output = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003eEvents in Namespace: {ns}\\u003c/h2\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/section\\u003e\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"--execution_id\",\"151\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b9842c1b-438d-410c-ad18-f01e0145433b/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # 주요 키워드 목록\\n    describe_keywords = [\\\"CrashLoopBackOff\\\", \\\"OOMKilled\\\", \\\"Error\\\", \\\"Failed\\\", \\\"Back-off\\\"]\\n    log_keywords = [\\\"Traceback\\\", \\\"panic\\\", \\\"exception\\\", \\\"fail\\\", \\\"error\\\"]\\n    event_keywords = [\\\"Failed\\\", \\\"BackOff\\\", \\\"Unhealthy\\\", \\\"Killing\\\", \\\"Evicted\\\"]\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    def match_keywords(lines, keywords):\\n        return \\\"\\\\n\\\".join([line for line in lines if any(k in line for k in keywords)])\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        # HTML header\\n        f.write(\\\"\\\"\\\"\\n        \\u003chtml\\u003e\\n        \\u003chead\\u003e\\n            \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n            \\u003cstyle\\u003e\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\n                h1 { color: #2c3e50; }\\n                h2 { color: #34495e; }\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\n                section { margin-bottom: 30px; }\\n            \\u003c/style\\u003e\\n        \\u003c/head\\u003e\\n        \\u003cbody\\u003e\\n        \\u003ch1\\u003eKubernetes 장애 점검 리포트\\u003c/h1\\u003e\\n        \\\"\\\"\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe_output = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            log_output = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003ePod: {pod_name} (Namespace: {namespace})\\u003c/h2\\u003e\\\")\\n                if matched_describe:\\n                    f.write(\\\"\\u003ch3\\u003eDescribe (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_describe + \\\"\\u003c/pre\\u003e\\\")\\n                if matched_logs:\\n                    f.write(\\\"\\u003ch3\\u003eLogs (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_logs + \\\"\\u003c/pre\\u003e\\\")\\n                f.write(\\\"\\u003c/section\\u003e\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events_output = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003eEvents in Namespace: {ns}\\u003c/h2\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/section\\u003e\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-gmf4s/2025/05/08/pod-log-to-github-gmf4s-system-container-impl-1948391354"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"--execution_id\",\"151\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b9842c1b-438d-410c-ad18-f01e0145433b/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # 주요 키워드 목록\\n    describe_keywords = [\\\"CrashLoopBackOff\\\", \\\"OOMKilled\\\", \\\"Error\\\", \\\"Failed\\\", \\\"Back-off\\\"]\\n    log_keywords = [\\\"Traceback\\\", \\\"panic\\\", \\\"exception\\\", \\\"fail\\\", \\\"error\\\"]\\n    event_keywords = [\\\"Failed\\\", \\\"BackOff\\\", \\\"Unhealthy\\\", \\\"Killing\\\", \\\"Evicted\\\"]\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    def match_keywords(lines, keywords):\\n        return \\\"\\\\n\\\".join([line for line in lines if any(k in line for k in keywords)])\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        # HTML header\\n        f.write(\\\"\\\"\\\"\\n        \\u003chtml\\u003e\\n        \\u003chead\\u003e\\n            \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n            \\u003cstyle\\u003e\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\n                h1 { color: #2c3e50; }\\n                h2 { color: #34495e; }\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\n                section { margin-bottom: 30px; }\\n            \\u003c/style\\u003e\\n        \\u003c/head\\u003e\\n        \\u003cbody\\u003e\\n        \\u003ch1\\u003eKubernetes 장애 점검 리포트\\u003c/h1\\u003e\\n        \\\"\\\"\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe_output = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            log_output = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003ePod: {pod_name} (Namespace: {namespace})\\u003c/h2\\u003e\\\")\\n                if matched_describe:\\n                    f.write(\\\"\\u003ch3\\u003eDescribe (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_describe + \\\"\\u003c/pre\\u003e\\\")\\n                if matched_logs:\\n                    f.write(\\\"\\u003ch3\\u003eLogs (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_logs + \\\"\\u003c/pre\\u003e\\\")\\n                f.write(\\\"\\u003c/section\\u003e\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events_output = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003eEvents in Namespace: {ns}\\u003c/h2\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/section\\u003e\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"--execution_id\",\"151\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b9842c1b-438d-410c-ad18-f01e0145433b/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # 주요 키워드 목록\\n    describe_keywords = [\\\"CrashLoopBackOff\\\", \\\"OOMKilled\\\", \\\"Error\\\", \\\"Failed\\\", \\\"Back-off\\\"]\\n    log_keywords = [\\\"Traceback\\\", \\\"panic\\\", \\\"exception\\\", \\\"fail\\\", \\\"error\\\"]\\n    event_keywords = [\\\"Failed\\\", \\\"BackOff\\\", \\\"Unhealthy\\\", \\\"Killing\\\", \\\"Evicted\\\"]\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    def match_keywords(lines, keywords):\\n        return \\\"\\\\n\\\".join([line for line in lines if any(k in line for k in keywords)])\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        # HTML header\\n        f.write(\\\"\\\"\\\"\\n        \\u003chtml\\u003e\\n        \\u003chead\\u003e\\n            \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n            \\u003cstyle\\u003e\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\n                h1 { color: #2c3e50; }\\n                h2 { color: #34495e; }\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\n                section { margin-bottom: 30px; }\\n            \\u003c/style\\u003e\\n        \\u003c/head\\u003e\\n        \\u003cbody\\u003e\\n        \\u003ch1\\u003eKubernetes 장애 점검 리포트\\u003c/h1\\u003e\\n        \\\"\\\"\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe_output = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            log_output = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003ePod: {pod_name} (Namespace: {namespace})\\u003c/h2\\u003e\\\")\\n                if matched_describe:\\n                    f.write(\\\"\\u003ch3\\u003eDescribe (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_describe + \\\"\\u003c/pre\\u003e\\\")\\n                if matched_logs:\\n                    f.write(\\\"\\u003ch3\\u003eLogs (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_logs + \\\"\\u003c/pre\\u003e\\\")\\n                f.write(\\\"\\u003c/section\\u003e\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events_output = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003eEvents in Namespace: {ns}\\u003c/h2\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/section\\u003e\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-gmf4s/2025/05/08/pod-log-to-github-gmf4s-system-container-impl-1948391354"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"--execution_id\",\"151\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b9842c1b-438d-410c-ad18-f01e0145433b/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # 주요 키워드 목록\\n    describe_keywords = [\\\"CrashLoopBackOff\\\", \\\"OOMKilled\\\", \\\"Error\\\", \\\"Failed\\\", \\\"Back-off\\\"]\\n    log_keywords = [\\\"Traceback\\\", \\\"panic\\\", \\\"exception\\\", \\\"fail\\\", \\\"error\\\"]\\n    event_keywords = [\\\"Failed\\\", \\\"BackOff\\\", \\\"Unhealthy\\\", \\\"Killing\\\", \\\"Evicted\\\"]\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    def match_keywords(lines, keywords):\\n        return \\\"\\\\n\\\".join([line for line in lines if any(k in line for k in keywords)])\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        # HTML header\\n        f.write(\\\"\\\"\\\"\\n        \\u003chtml\\u003e\\n        \\u003chead\\u003e\\n            \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n            \\u003cstyle\\u003e\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\n                h1 { color: #2c3e50; }\\n                h2 { color: #34495e; }\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\n                section { margin-bottom: 30px; }\\n            \\u003c/style\\u003e\\n        \\u003c/head\\u003e\\n        \\u003cbody\\u003e\\n        \\u003ch1\\u003eKubernetes 장애 점검 리포트\\u003c/h1\\u003e\\n        \\\"\\\"\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe_output = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            log_output = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003ePod: {pod_name} (Namespace: {namespace})\\u003c/h2\\u003e\\\")\\n                if matched_describe:\\n                    f.write(\\\"\\u003ch3\\u003eDescribe (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_describe + \\\"\\u003c/pre\\u003e\\\")\\n                if matched_logs:\\n                    f.write(\\\"\\u003ch3\\u003eLogs (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_logs + \\\"\\u003c/pre\\u003e\\\")\\n                f.write(\\\"\\u003c/section\\u003e\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events_output = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003eEvents in Namespace: {ns}\\u003c/h2\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/section\\u003e\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
          describe_keywords = ["CrashLoopBackOff", "OOMKilled", "Error", "Failed", "Back-off"]
          event_keywords = ["Failed", "BackOff", "Unhealthy", "Killing", "Evicted"]
                  except ValueError:
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"--execution_id\",\"151\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b9842c1b-438d-410c-ad18-f01e0145433b/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # 주요 키워드 목록\\n    describe_keywords = [\\\"CrashLoopBackOff\\\", \\\"OOMKilled\\\", \\\"Error\\\", \\\"Failed\\\", \\\"Back-off\\\"]\\n    log_keywords = [\\\"Traceback\\\", \\\"panic\\\", \\\"exception\\\", \\\"fail\\\", \\\"error\\\"]\\n    event_keywords = [\\\"Failed\\\", \\\"BackOff\\\", \\\"Unhealthy\\\", \\\"Killing\\\", \\\"Evicted\\\"]\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    def match_keywords(lines, keywords):\\n        return \\\"\\\\n\\\".join([line for line in lines if any(k in line for k in keywords)])\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        # HTML header\\n        f.write(\\\"\\\"\\\"\\n        \\u003chtml\\u003e\\n        \\u003chead\\u003e\\n            \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n            \\u003cstyle\\u003e\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\n                h1 { color: #2c3e50; }\\n                h2 { color: #34495e; }\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\n                section { margin-bottom: 30px; }\\n            \\u003c/style\\u003e\\n        \\u003c/head\\u003e\\n        \\u003cbody\\u003e\\n        \\u003ch1\\u003eKubernetes 장애 점검 리포트\\u003c/h1\\u003e\\n        \\\"\\\"\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe_output = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            log_output = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003ePod: {pod_name} (Namespace: {namespace})\\u003c/h2\\u003e\\\")\\n                if matched_describe:\\n                    f.write(\\\"\\u003ch3\\u003eDescribe (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_describe + \\\"\\u003c/pre\\u003e\\\")\\n                if matched_logs:\\n                    f.write(\\\"\\u003ch3\\u003eLogs (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_logs + \\\"\\u003c/pre\\u003e\\\")\\n                f.write(\\\"\\u003c/section\\u003e\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events_output = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003eEvents in Namespace: {ns}\\u003c/h2\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/section\\u003e\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-gmf4s/2025/05/08/pod-log-to-github-gmf4s-system-container-impl-1948391354"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"--execution_id\",\"151\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b9842c1b-438d-410c-ad18-f01e0145433b/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # 주요 키워드 목록\\n    describe_keywords = [\\\"CrashLoopBackOff\\\", \\\"OOMKilled\\\", \\\"Error\\\", \\\"Failed\\\", \\\"Back-off\\\"]\\n    log_keywords = [\\\"Traceback\\\", \\\"panic\\\", \\\"exception\\\", \\\"fail\\\", \\\"error\\\"]\\n    event_keywords = [\\\"Failed\\\", \\\"BackOff\\\", \\\"Unhealthy\\\", \\\"Killing\\\", \\\"Evicted\\\"]\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    def match_keywords(lines, keywords):\\n        return \\\"\\\\n\\\".join([line for line in lines if any(k in line for k in keywords)])\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        # HTML header\\n        f.write(\\\"\\\"\\\"\\n        \\u003chtml\\u003e\\n        \\u003chead\\u003e\\n            \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n            \\u003cstyle\\u003e\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\n                h1 { color: #2c3e50; }\\n                h2 { color: #34495e; }\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\n                section { margin-bottom: 30px; }\\n            \\u003c/style\\u003e\\n        \\u003c/head\\u003e\\n        \\u003cbody\\u003e\\n        \\u003ch1\\u003eKubernetes 장애 점검 리포트\\u003c/h1\\u003e\\n        \\\"\\\"\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe_output = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            log_output = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003ePod: {pod_name} (Namespace: {namespace})\\u003c/h2\\u003e\\\")\\n                if matched_describe:\\n                    f.write(\\\"\\u003ch3\\u003eDescribe (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_describe + \\\"\\u003c/pre\\u003e\\\")\\n                if matched_logs:\\n                    f.write(\\\"\\u003ch3\\u003eLogs (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_logs + \\\"\\u003c/pre\\u003e\\\")\\n                f.write(\\\"\\u003c/section\\u003e\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events_output = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003eEvents in Namespace: {ns}\\u003c/h2\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/section\\u003e\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}

Pod: pod-log-to-github-gmf4s-system-dag-driver-2252566396 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T06:35:47.318Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-kcp9j-system-container-driver-1719766903 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"93"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","ac3a4dbc-00a5-4dfd-893b-69388ca0e742","--dag_execution_id","93","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-kcp9j/2025/05/08/pod-log-to-github-kcp9j-system-container-driver-1719766903"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"93"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","ac3a4dbc-00a5-4dfd-893b-69388ca0e742","--dag_execution_id","93","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-kcp9j/2025/05/08/pod-log-to-github-kcp9j-system-container-driver-1719766903"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str = \"main\"\n):\n    import os\n    import subprocess\n\n\n    report_path = report_file.path\n    print(\"Checking if report exists at:\", report_path)\n    if not os.path.exists(report_path):\n        raise FileNotFoundError(f\"{report_path} does not exist\")\n\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    subprocess.run([\"cp\", report_path, os.path.join(repo_dir, \"report.md\")], check=True)\n\n    # Git user config\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", \"Add report\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(\"Report committed and pushed to GitHub.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"93"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","ac3a4dbc-00a5-4dfd-893b-69388ca0e742","--dag_execution_id","93","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-kcp9j/2025/05/08/pod-log-to-github-kcp9j-system-container-driver-1719766903"}}}

Logs (filtered)

time="2025-05-08T01:05:24.337Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-kcp9j-system-container-driver-2792783052 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T01:04:05.427Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-kcp9j-system-container-impl-1592955769 (Namespace: kubeflow-user-example-com)

Describe (filtered)

Status:       Failed
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"ac3a4dbc-00a5-4dfd-893b-69388ca0e742\",\"--execution_id\",\"95\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"9\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ac3a4dbc-00a5-4dfd-893b-69388ca0e742/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-kcp9j/2025/05/08/pod-log-to-github-kcp9j-system-container-impl-1592955769"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"ac3a4dbc-00a5-4dfd-893b-69388ca0e742\",\"--execution_id\",\"95\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"9\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ac3a4dbc-00a5-4dfd-893b-69388ca0e742/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"ac3a4dbc-00a5-4dfd-893b-69388ca0e742\",\"--execution_id\",\"95\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"9\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ac3a4dbc-00a5-4dfd-893b-69388ca0e742/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-kcp9j/2025/05/08/pod-log-to-github-kcp9j-system-container-impl-1592955769"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"ac3a4dbc-00a5-4dfd-893b-69388ca0e742\",\"--execution_id\",\"95\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"9\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ac3a4dbc-00a5-4dfd-893b-69388ca0e742/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"ac3a4dbc-00a5-4dfd-893b-69388ca0e742\",\"--execution_id\",\"95\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"9\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ac3a4dbc-00a5-4dfd-893b-69388ca0e742/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-kcp9j/2025/05/08/pod-log-to-github-kcp9j-system-container-impl-1592955769"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"ac3a4dbc-00a5-4dfd-893b-69388ca0e742\",\"--execution_id\",\"95\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"9\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ac3a4dbc-00a5-4dfd-893b-69388ca0e742/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
              raise FileNotFoundError(f"{report_path} does not exist")
      Reason:       Error
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"ac3a4dbc-00a5-4dfd-893b-69388ca0e742\",\"--execution_id\",\"95\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"9\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ac3a4dbc-00a5-4dfd-893b-69388ca0e742/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-kcp9j/2025/05/08/pod-log-to-github-kcp9j-system-container-impl-1592955769"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"ac3a4dbc-00a5-4dfd-893b-69388ca0e742\",\"--execution_id\",\"95\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"9\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ac3a4dbc-00a5-4dfd-893b-69388ca0e742/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}

Logs (filtered)

remote: error: GH013: Repository rule violations found for refs/heads/main.        
error: failed to push some refs to 'https://github.com/Rosa1026/Kubeflow.git'
Traceback (most recent call last):
F0508 01:05:44.575061      23 main.go:49] failed to execute component: exit status 1
time="2025-05-08T01:05:45.451Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T01:05:46.398Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-kcp9j-system-container-impl-1691535522 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T01:05:10.700Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T01:05:11.629Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-kcp9j-system-dag-driver-426681894 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T01:03:55.354Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-kmxkl-system-container-driver-3811604332 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import subprocess\\n    import shutil\\n    import json\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.md\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"# 장애 복기 리포트 (keyword: `{keyword}`)\\\\n\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\\\n\\\")\\n                if matched_describe:\\n                    f.write(\\\"### describe\\\\n```text\\\\n\\\" + matched_describe + \\\"\\\\n```\\\\n\\\")\\n                if matched_logs:\\n                    f.write(\\\"### logs\\\\n```text\\\\n\\\" + matched_logs + \\\"\\\\n```\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\\\n## Events in Namespace: `{ns}`\\\\n```text\\\\n{events}\\\\n```\\\\n\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"102"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","b154ee1c-cfaf-4508-8b9b-ac90af9ada50","--dag_execution_id","102","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import subprocess\\n    import shutil\\n    import json\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.md\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"# 장애 복기 리포트 (keyword: `{keyword}`)\\\\n\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\\\n\\\")\\n                if matched_describe:\\n                    f.write(\\\"### describe\\\\n```text\\\\n\\\" + matched_describe + \\\"\\\\n```\\\\n\\\")\\n                if matched_logs:\\n                    f.write(\\\"### logs\\\\n```text\\\\n\\\" + matched_logs + \\\"\\\\n```\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\\\n## Events in Namespace: `{ns}`\\\\n```text\\\\n{events}\\\\n```\\\\n\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-kmxkl/2025/05/08/pod-log-to-github-kmxkl-system-container-driver-3811604332"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import subprocess\\n    import shutil\\n    import json\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.md\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"# 장애 복기 리포트 (keyword: `{keyword}`)\\\\n\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\\\n\\\")\\n                if matched_describe:\\n                    f.write(\\\"### describe\\\\n```text\\\\n\\\" + matched_describe + \\\"\\\\n```\\\\n\\\")\\n                if matched_logs:\\n                    f.write(\\\"### logs\\\\n```text\\\\n\\\" + matched_logs + \\\"\\\\n```\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\\\n## Events in Namespace: `{ns}`\\\\n```text\\\\n{events}\\\\n```\\\\n\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"102"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","b154ee1c-cfaf-4508-8b9b-ac90af9ada50","--dag_execution_id","102","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import subprocess\\n    import shutil\\n    import json\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.md\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"# 장애 복기 리포트 (keyword: `{keyword}`)\\\\n\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\\\n\\\")\\n                if matched_describe:\\n                    f.write(\\\"### describe\\\\n```text\\\\n\\\" + matched_describe + \\\"\\\\n```\\\\n\\\")\\n                if matched_logs:\\n                    f.write(\\\"### logs\\\\n```text\\\\n\\\" + matched_logs + \\\"\\\\n```\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\\\n## Events in Namespace: `{ns}`\\\\n```text\\\\n{events}\\\\n```\\\\n\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-kmxkl/2025/05/08/pod-log-to-github-kmxkl-system-container-driver-3811604332"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","collect_filtered_logs"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \"crash\"):\n    import subprocess\n    import shutil\n    import json\n    import os\n\n    tmp_report_path = \"/tmp/report.md\"\n    seen_namespaces = set()\n    namespace_events = {}\n\n    # kubectl 설치 방법 변경\n    subprocess.run([\"apt-get\", \"update\", \"-y\"], check=True)\n    subprocess.run([\n        \"curl\", \"-LO\", \"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\"\n    ], check=True)\n    subprocess.run([\"chmod\", \"+x\", \"kubectl\"], check=True)\n    subprocess.run([\"mv\", \"kubectl\", \"/usr/local/bin/\"], check=True)\n\n    # Get all pods in all namespaces\n    pods_output = subprocess.getoutput(\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\")\n    pod_lines = pods_output.strip().split(\"\\n\")\n\n    with open(tmp_report_path, \"w\") as f:\n        f.write(f\"# 장애 복기 리포트 (keyword: `{keyword}`)\\n\\n\")\n\n        for line in pod_lines:\n            if not line.strip():\n                continue\n            try:\n                namespace, pod_name = line.split()\n            except ValueError:\n                continue\n\n            describe = subprocess.getoutput(f\"kubectl describe pod {pod_name} -n {namespace}\")\n            logs = subprocess.getoutput(f\"kubectl logs {pod_name} -n {namespace}\")\n\n            matched_describe = \"\\n\".join([l for l in describe.splitlines() if keyword in l])\n            matched_logs = \"\\n\".join([l for l in logs.splitlines() if keyword in l])\n\n            if matched_describe or matched_logs:\n                f.write(f\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\n\")\n                if matched_describe:\n                    f.write(\"### describe\\n```text\\n\" + matched_describe + \"\\n```\\n\")\n                if matched_logs:\n                    f.write(\"### logs\\n```text\\n\" + matched_logs + \"\\n```\\n\")\n\n            if namespace not in seen_namespaces:\n                events = subprocess.getoutput(f\"kubectl get events -n {namespace}\")\n                matched_events = \"\\n\".join([l for l in events.splitlines() if keyword in l])\n                if matched_events:\n                    namespace_events[namespace] = matched_events\n                seen_namespaces.add(namespace)\n\n        # Write filtered events grouped by namespace\n        for ns, events in namespace_events.items():\n            f.write(f\"\\n## Events in Namespace: `{ns}`\\n```text\\n{events}\\n```\\n\")\n\n    subprocess.run([\"cp\", tmp_report_path, output_report.path], check=True)\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import subprocess\\n    import shutil\\n    import json\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.md\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"# 장애 복기 리포트 (keyword: `{keyword}`)\\\\n\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\\\n\\\")\\n                if matched_describe:\\n                    f.write(\\\"### describe\\\\n```text\\\\n\\\" + matched_describe + \\\"\\\\n```\\\\n\\\")\\n                if matched_logs:\\n                    f.write(\\\"### logs\\\\n```text\\\\n\\\" + matched_logs + \\\"\\\\n```\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\\\n## Events in Namespace: `{ns}`\\\\n```text\\\\n{events}\\\\n```\\\\n\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"102"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","b154ee1c-cfaf-4508-8b9b-ac90af9ada50","--dag_execution_id","102","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import subprocess\\n    import shutil\\n    import json\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.md\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"# 장애 복기 리포트 (keyword: `{keyword}`)\\\\n\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\\\n\\\")\\n                if matched_describe:\\n                    f.write(\\\"### describe\\\\n```text\\\\n\\\" + matched_describe + \\\"\\\\n```\\\\n\\\")\\n                if matched_logs:\\n                    f.write(\\\"### logs\\\\n```text\\\\n\\\" + matched_logs + \\\"\\\\n```\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\\\n## Events in Namespace: `{ns}`\\\\n```text\\\\n{events}\\\\n```\\\\n\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-kmxkl/2025/05/08/pod-log-to-github-kmxkl-system-container-driver-3811604332"}}}

Logs (filtered)

time="2025-05-08T02:00:17.112Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-kmxkl-system-container-driver-4102728141 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"102"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","b154ee1c-cfaf-4508-8b9b-ac90af9ada50","--dag_execution_id","102","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-kmxkl/2025/05/08/pod-log-to-github-kmxkl-system-container-driver-4102728141"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"102"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","b154ee1c-cfaf-4508-8b9b-ac90af9ada50","--dag_execution_id","102","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-kmxkl/2025/05/08/pod-log-to-github-kmxkl-system-container-driver-4102728141"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str = \"main\"\n):\n    import os\n    import subprocess\n\n\n    report_path = report_file.path\n    print(\"Checking if report exists at:\", report_path)\n    if not os.path.exists(report_path):\n        raise FileNotFoundError(f\"{report_path} does not exist\")\n\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    subprocess.run([\"cp\", report_path, os.path.join(repo_dir, \"report.md\")], check=True)\n\n    # Git user config\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", \"Add report\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(\"Report committed and pushed to GitHub.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"102"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","b154ee1c-cfaf-4508-8b9b-ac90af9ada50","--dag_execution_id","102","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-kmxkl/2025/05/08/pod-log-to-github-kmxkl-system-container-driver-4102728141"}}}

Logs (filtered)

time="2025-05-08T02:01:47.021Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-kmxkl-system-container-impl-2051720987 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b154ee1c-cfaf-4508-8b9b-ac90af9ada50\",\"--execution_id\",\"104\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"12\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b154ee1c-cfaf-4508-8b9b-ac90af9ada50/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-kmxkl/2025/05/08/pod-log-to-github-kmxkl-system-container-impl-2051720987"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b154ee1c-cfaf-4508-8b9b-ac90af9ada50\",\"--execution_id\",\"104\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"12\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b154ee1c-cfaf-4508-8b9b-ac90af9ada50/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b154ee1c-cfaf-4508-8b9b-ac90af9ada50\",\"--execution_id\",\"104\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"12\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b154ee1c-cfaf-4508-8b9b-ac90af9ada50/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-kmxkl/2025/05/08/pod-log-to-github-kmxkl-system-container-impl-2051720987"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b154ee1c-cfaf-4508-8b9b-ac90af9ada50\",\"--execution_id\",\"104\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"12\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b154ee1c-cfaf-4508-8b9b-ac90af9ada50/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b154ee1c-cfaf-4508-8b9b-ac90af9ada50\",\"--execution_id\",\"104\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"12\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b154ee1c-cfaf-4508-8b9b-ac90af9ada50/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-kmxkl/2025/05/08/pod-log-to-github-kmxkl-system-container-impl-2051720987"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b154ee1c-cfaf-4508-8b9b-ac90af9ada50\",\"--execution_id\",\"104\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"12\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b154ee1c-cfaf-4508-8b9b-ac90af9ada50/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
              raise FileNotFoundError(f"{report_path} does not exist")
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b154ee1c-cfaf-4508-8b9b-ac90af9ada50\",\"--execution_id\",\"104\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"12\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b154ee1c-cfaf-4508-8b9b-ac90af9ada50/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-kmxkl/2025/05/08/pod-log-to-github-kmxkl-system-container-impl-2051720987"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b154ee1c-cfaf-4508-8b9b-ac90af9ada50\",\"--execution_id\",\"104\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"12\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b154ee1c-cfaf-4508-8b9b-ac90af9ada50/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}

Logs (filtered)

time="2025-05-08T02:02:06.228Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T02:02:07.171Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-kmxkl-system-container-impl-2099142466 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b154ee1c-cfaf-4508-8b9b-ac90af9ada50\",\"--execution_id\",\"103\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b154ee1c-cfaf-4508-8b9b-ac90af9ada50/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import subprocess\\n    import shutil\\n    import json\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.md\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"# 장애 복기 리포트 (keyword: `{keyword}`)\\\\n\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\\\n\\\")\\n                if matched_describe:\\n                    f.write(\\\"### describe\\\\n```text\\\\n\\\" + matched_describe + \\\"\\\\n```\\\\n\\\")\\n                if matched_logs:\\n                    f.write(\\\"### logs\\\\n```text\\\\n\\\" + matched_logs + \\\"\\\\n```\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\\\n## Events in Namespace: `{ns}`\\\\n```text\\\\n{events}\\\\n```\\\\n\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-kmxkl/2025/05/08/pod-log-to-github-kmxkl-system-container-impl-2099142466"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b154ee1c-cfaf-4508-8b9b-ac90af9ada50\",\"--execution_id\",\"103\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b154ee1c-cfaf-4508-8b9b-ac90af9ada50/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import subprocess\\n    import shutil\\n    import json\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.md\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"# 장애 복기 리포트 (keyword: `{keyword}`)\\\\n\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\\\n\\\")\\n                if matched_describe:\\n                    f.write(\\\"### describe\\\\n```text\\\\n\\\" + matched_describe + \\\"\\\\n```\\\\n\\\")\\n                if matched_logs:\\n                    f.write(\\\"### logs\\\\n```text\\\\n\\\" + matched_logs + \\\"\\\\n```\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\\\n## Events in Namespace: `{ns}`\\\\n```text\\\\n{events}\\\\n```\\\\n\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b154ee1c-cfaf-4508-8b9b-ac90af9ada50\",\"--execution_id\",\"103\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b154ee1c-cfaf-4508-8b9b-ac90af9ada50/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import subprocess\\n    import shutil\\n    import json\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.md\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"# 장애 복기 리포트 (keyword: `{keyword}`)\\\\n\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\\\n\\\")\\n                if matched_describe:\\n                    f.write(\\\"### describe\\\\n```text\\\\n\\\" + matched_describe + \\\"\\\\n```\\\\n\\\")\\n                if matched_logs:\\n                    f.write(\\\"### logs\\\\n```text\\\\n\\\" + matched_logs + \\\"\\\\n```\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\\\n## Events in Namespace: `{ns}`\\\\n```text\\\\n{events}\\\\n```\\\\n\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-kmxkl/2025/05/08/pod-log-to-github-kmxkl-system-container-impl-2099142466"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b154ee1c-cfaf-4508-8b9b-ac90af9ada50\",\"--execution_id\",\"103\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b154ee1c-cfaf-4508-8b9b-ac90af9ada50/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import subprocess\\n    import shutil\\n    import json\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.md\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"# 장애 복기 리포트 (keyword: `{keyword}`)\\\\n\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\\\n\\\")\\n                if matched_describe:\\n                    f.write(\\\"### describe\\\\n```text\\\\n\\\" + matched_describe + \\\"\\\\n```\\\\n\\\")\\n                if matched_logs:\\n                    f.write(\\\"### logs\\\\n```text\\\\n\\\" + matched_logs + \\\"\\\\n```\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\\\n## Events in Namespace: `{ns}`\\\\n```text\\\\n{events}\\\\n```\\\\n\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b154ee1c-cfaf-4508-8b9b-ac90af9ada50\",\"--execution_id\",\"103\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b154ee1c-cfaf-4508-8b9b-ac90af9ada50/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import subprocess\\n    import shutil\\n    import json\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.md\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"# 장애 복기 리포트 (keyword: `{keyword}`)\\\\n\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\\\n\\\")\\n                if matched_describe:\\n                    f.write(\\\"### describe\\\\n```text\\\\n\\\" + matched_describe + \\\"\\\\n```\\\\n\\\")\\n                if matched_logs:\\n                    f.write(\\\"### logs\\\\n```text\\\\n\\\" + matched_logs + \\\"\\\\n```\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\\\n## Events in Namespace: `{ns}`\\\\n```text\\\\n{events}\\\\n```\\\\n\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-kmxkl/2025/05/08/pod-log-to-github-kmxkl-system-container-impl-2099142466"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b154ee1c-cfaf-4508-8b9b-ac90af9ada50\",\"--execution_id\",\"103\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b154ee1c-cfaf-4508-8b9b-ac90af9ada50/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import subprocess\\n    import shutil\\n    import json\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.md\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"# 장애 복기 리포트 (keyword: `{keyword}`)\\\\n\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\\\n\\\")\\n                if matched_describe:\\n                    f.write(\\\"### describe\\\\n```text\\\\n\\\" + matched_describe + \\\"\\\\n```\\\\n\\\")\\n                if matched_logs:\\n                    f.write(\\\"### logs\\\\n```text\\\\n\\\" + matched_logs + \\\"\\\\n```\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\\\n## Events in Namespace: `{ns}`\\\\n```text\\\\n{events}\\\\n```\\\\n\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
                  except ValueError:
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b154ee1c-cfaf-4508-8b9b-ac90af9ada50\",\"--execution_id\",\"103\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b154ee1c-cfaf-4508-8b9b-ac90af9ada50/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import subprocess\\n    import shutil\\n    import json\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.md\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"# 장애 복기 리포트 (keyword: `{keyword}`)\\\\n\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\\\n\\\")\\n                if matched_describe:\\n                    f.write(\\\"### describe\\\\n```text\\\\n\\\" + matched_describe + \\\"\\\\n```\\\\n\\\")\\n                if matched_logs:\\n                    f.write(\\\"### logs\\\\n```text\\\\n\\\" + matched_logs + \\\"\\\\n```\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\\\n## Events in Namespace: `{ns}`\\\\n```text\\\\n{events}\\\\n```\\\\n\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-kmxkl/2025/05/08/pod-log-to-github-kmxkl-system-container-impl-2099142466"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b154ee1c-cfaf-4508-8b9b-ac90af9ada50\",\"--execution_id\",\"103\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b154ee1c-cfaf-4508-8b9b-ac90af9ada50/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import subprocess\\n    import shutil\\n    import json\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.md\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"# 장애 복기 리포트 (keyword: `{keyword}`)\\\\n\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\\\n\\\")\\n                if matched_describe:\\n                    f.write(\\\"### describe\\\\n```text\\\\n\\\" + matched_describe + \\\"\\\\n```\\\\n\\\")\\n                if matched_logs:\\n                    f.write(\\\"### logs\\\\n```text\\\\n\\\" + matched_logs + \\\"\\\\n```\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\\\n## Events in Namespace: `{ns}`\\\\n```text\\\\n{events}\\\\n```\\\\n\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}

Logs (filtered)

time="2025-05-08T02:01:33.422Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T02:01:34.367Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-kmxkl-system-dag-driver-3954723572 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T02:00:06.940Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-ljqp8-system-container-driver-1813844290 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"119"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","9e2b32a7-be60-4916-8003-81ecba5290db","--dag_execution_id","119","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-ljqp8/2025/05/08/pod-log-to-github-ljqp8-system-container-driver-1813844290"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"119"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","9e2b32a7-be60-4916-8003-81ecba5290db","--dag_execution_id","119","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-ljqp8/2025/05/08/pod-log-to-github-ljqp8-system-container-driver-1813844290"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","collect_filtered_logs"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \"crash\"):\n    import shutil\n    import json\n    import subprocess\n    import os\n\n    tmp_report_path = \"/tmp/report.html\"\n    seen_namespaces = set()\n    namespace_events = {}\n\n    # kubectl 설치 방법 변경\n    subprocess.run([\"apt-get\", \"update\", \"-y\"], check=True)\n    subprocess.run([ \n        \"curl\", \"-LO\", \"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\"\n    ], check=True)\n    subprocess.run([\"chmod\", \"+x\", \"kubectl\"], check=True)\n    subprocess.run([\"mv\", \"kubectl\", \"/usr/local/bin/\"], check=True)\n\n    # Get all pods in all namespaces\n    pods_output = subprocess.getoutput(\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\")\n    pod_lines = pods_output.strip().split(\"\\n\")\n\n    with open(tmp_report_path, \"w\") as f:\n        f.write(f\"\u003chtml\u003e\u003cbody\u003e\u003ch1\u003e장애 복기 리포트 (keyword: `{keyword}`)\u003c/h1\u003e\\n\")\n\n        for line in pod_lines:\n            if not line.strip():\n                continue\n            try:\n                namespace, pod_name = line.split()\n            except ValueError:\n                continue\n\n            describe = subprocess.getoutput(f\"kubectl describe pod {pod_name} -n {namespace}\")\n            logs = subprocess.getoutput(f\"kubectl logs {pod_name} -n {namespace}\")\n\n            matched_describe = \"\\n\".join([l for l in describe.splitlines() if keyword in l])\n            matched_logs = \"\\n\".join([l for l in logs.splitlines() if keyword in l])\n\n            if matched_describe or matched_logs:\n                f.write(f\"\u003ch2\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\u003c/h2\u003e\\n\")\n                if matched_describe:\n                    f.write(f\"\u003ch3\u003eDescribe\u003c/h3\u003e\u003cpre\u003e{matched_describe}\u003c/pre\u003e\\n\")\n                if matched_logs:\n                    f.write(f\"\u003ch3\u003eLogs\u003c/h3\u003e\u003cpre\u003e{matched_logs}\u003c/pre\u003e\\n\")\n\n            if namespace not in seen_namespaces:\n                events = subprocess.getoutput(f\"kubectl get events -n {namespace}\")\n                matched_events = \"\\n\".join([l for l in events.splitlines() if keyword in l])\n                if matched_events:\n                    namespace_events[namespace] = matched_events\n                seen_namespaces.add(namespace)\n\n        # Write filtered events grouped by namespace\n        for ns, events in namespace_events.items():\n            f.write(f\"\u003ch3\u003eEvents in Namespace: `{ns}`\u003c/h3\u003e\u003cpre\u003e{events}\u003c/pre\u003e\\n\")\n\n        f.write(\"\u003c/body\u003e\u003c/html\u003e\")\n\n    subprocess.run([\"cp\", tmp_report_path, output_report.path], check=True)\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"119"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","9e2b32a7-be60-4916-8003-81ecba5290db","--dag_execution_id","119","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-ljqp8/2025/05/08/pod-log-to-github-ljqp8-system-container-driver-1813844290"}}}

Logs (filtered)

time="2025-05-08T05:05:43.121Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-ljqp8-system-container-driver-3009813527 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"119"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","9e2b32a7-be60-4916-8003-81ecba5290db","--dag_execution_id","119","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-ljqp8/2025/05/08/pod-log-to-github-ljqp8-system-container-driver-3009813527"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"119"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","9e2b32a7-be60-4916-8003-81ecba5290db","--dag_execution_id","119","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-ljqp8/2025/05/08/pod-log-to-github-ljqp8-system-container-driver-3009813527"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str = \"main\"\n):\n\n    import subprocess\n    import shutil\n    import os\n    from datetime import datetime\n\n    report_path = report_file.path\n    print(\"Checking if report exists at:\", report_path)\n    if not os.path.exists(report_path):\n        raise FileNotFoundError(f\"{report_path} does not exist\")\n\n    # HTML 파일 이름 만들기\n    timestamp = datetime.now().strftime(\"report-%Y%m%d-%H%M%S.html\")\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n    report_target_path = os.path.join(repo_dir, timestamp)\n\n    # Git clone\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    shutil.copyfile(report_path, report_target_path)\n\n    # Git config\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    # Git commit \u0026 push\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", f\"Add {timestamp}\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(f\"Report committed and pushed to GitHub as {timestamp}.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"119"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","9e2b32a7-be60-4916-8003-81ecba5290db","--dag_execution_id","119","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-ljqp8/2025/05/08/pod-log-to-github-ljqp8-system-container-driver-3009813527"}}}

Logs (filtered)

time="2025-05-08T05:07:18.217Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-ljqp8-system-container-impl-1421462284 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"9e2b32a7-be60-4916-8003-81ecba5290db\",\"--execution_id\",\"120\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/9e2b32a7-be60-4916-8003-81ecba5290db/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-ljqp8/2025/05/08/pod-log-to-github-ljqp8-system-container-impl-1421462284"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"9e2b32a7-be60-4916-8003-81ecba5290db\",\"--execution_id\",\"120\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/9e2b32a7-be60-4916-8003-81ecba5290db/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"9e2b32a7-be60-4916-8003-81ecba5290db\",\"--execution_id\",\"120\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/9e2b32a7-be60-4916-8003-81ecba5290db/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-ljqp8/2025/05/08/pod-log-to-github-ljqp8-system-container-impl-1421462284"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"9e2b32a7-be60-4916-8003-81ecba5290db\",\"--execution_id\",\"120\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/9e2b32a7-be60-4916-8003-81ecba5290db/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"9e2b32a7-be60-4916-8003-81ecba5290db\",\"--execution_id\",\"120\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/9e2b32a7-be60-4916-8003-81ecba5290db/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-ljqp8/2025/05/08/pod-log-to-github-ljqp8-system-container-impl-1421462284"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"9e2b32a7-be60-4916-8003-81ecba5290db\",\"--execution_id\",\"120\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/9e2b32a7-be60-4916-8003-81ecba5290db/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
                  except ValueError:
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"9e2b32a7-be60-4916-8003-81ecba5290db\",\"--execution_id\",\"120\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/9e2b32a7-be60-4916-8003-81ecba5290db/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-ljqp8/2025/05/08/pod-log-to-github-ljqp8-system-container-impl-1421462284"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"9e2b32a7-be60-4916-8003-81ecba5290db\",\"--execution_id\",\"120\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/9e2b32a7-be60-4916-8003-81ecba5290db/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}

Logs (filtered)

time="2025-05-08T05:07:04.380Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T05:07:05.321Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-ljqp8-system-container-impl-4244329241 (Namespace: kubeflow-user-example-com)

Describe (filtered)

Status:       Failed
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"9e2b32a7-be60-4916-8003-81ecba5290db\",\"--execution_id\",\"121\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"17\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/9e2b32a7-be60-4916-8003-81ecba5290db/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-ljqp8/2025/05/08/pod-log-to-github-ljqp8-system-container-impl-4244329241"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"9e2b32a7-be60-4916-8003-81ecba5290db\",\"--execution_id\",\"121\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"17\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/9e2b32a7-be60-4916-8003-81ecba5290db/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"9e2b32a7-be60-4916-8003-81ecba5290db\",\"--execution_id\",\"121\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"17\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/9e2b32a7-be60-4916-8003-81ecba5290db/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-ljqp8/2025/05/08/pod-log-to-github-ljqp8-system-container-impl-4244329241"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"9e2b32a7-be60-4916-8003-81ecba5290db\",\"--execution_id\",\"121\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"17\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/9e2b32a7-be60-4916-8003-81ecba5290db/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"9e2b32a7-be60-4916-8003-81ecba5290db\",\"--execution_id\",\"121\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"17\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/9e2b32a7-be60-4916-8003-81ecba5290db/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-ljqp8/2025/05/08/pod-log-to-github-ljqp8-system-container-impl-4244329241"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"9e2b32a7-be60-4916-8003-81ecba5290db\",\"--execution_id\",\"121\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"17\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/9e2b32a7-be60-4916-8003-81ecba5290db/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
              raise FileNotFoundError(f"{report_path} does not exist")
      Reason:       Error
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"9e2b32a7-be60-4916-8003-81ecba5290db\",\"--execution_id\",\"121\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"17\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/9e2b32a7-be60-4916-8003-81ecba5290db/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-ljqp8/2025/05/08/pod-log-to-github-ljqp8-system-container-impl-4244329241"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"9e2b32a7-be60-4916-8003-81ecba5290db\",\"--execution_id\",\"121\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"17\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/9e2b32a7-be60-4916-8003-81ecba5290db/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}

Logs (filtered)

Traceback (most recent call last):
F0508 05:07:37.009291      24 main.go:49] failed to execute component: exit status 1
time="2025-05-08T05:07:37.366Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T05:07:38.318Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-ljqp8-system-dag-driver-1177292486 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T05:05:32.926Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-lm5jl-system-container-driver-1739908828 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"96"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","744b38f8-98d0-4d54-be2b-8b7a1c644200","--dag_execution_id","96","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-lm5jl/2025/05/08/pod-log-to-github-lm5jl-system-container-driver-1739908828"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"96"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","744b38f8-98d0-4d54-be2b-8b7a1c644200","--dag_execution_id","96","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-lm5jl/2025/05/08/pod-log-to-github-lm5jl-system-container-driver-1739908828"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str = \"main\"\n):\n    import os\n    import subprocess\n\n\n    report_path = report_file.path\n    print(\"Checking if report exists at:\", report_path)\n    if not os.path.exists(report_path):\n        raise FileNotFoundError(f\"{report_path} does not exist\")\n\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    subprocess.run([\"cp\", report_path, os.path.join(repo_dir, \"report.md\")], check=True)\n\n    # Git user config\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", \"Add report\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(\"Report committed and pushed to GitHub.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"96"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","744b38f8-98d0-4d54-be2b-8b7a1c644200","--dag_execution_id","96","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-lm5jl/2025/05/08/pod-log-to-github-lm5jl-system-container-driver-1739908828"}}}

Logs (filtered)

time="2025-05-08T01:11:10.628Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-lm5jl-system-container-driver-3976342479 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T01:09:47.747Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-lm5jl-system-container-impl-3089604882 (Namespace: kubeflow-user-example-com)

Describe (filtered)

Status:       Failed
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"744b38f8-98d0-4d54-be2b-8b7a1c644200\",\"--execution_id\",\"98\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"10\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/744b38f8-98d0-4d54-be2b-8b7a1c644200/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-lm5jl/2025/05/08/pod-log-to-github-lm5jl-system-container-impl-3089604882"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"744b38f8-98d0-4d54-be2b-8b7a1c644200\",\"--execution_id\",\"98\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"10\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/744b38f8-98d0-4d54-be2b-8b7a1c644200/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"744b38f8-98d0-4d54-be2b-8b7a1c644200\",\"--execution_id\",\"98\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"10\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/744b38f8-98d0-4d54-be2b-8b7a1c644200/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-lm5jl/2025/05/08/pod-log-to-github-lm5jl-system-container-impl-3089604882"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"744b38f8-98d0-4d54-be2b-8b7a1c644200\",\"--execution_id\",\"98\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"10\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/744b38f8-98d0-4d54-be2b-8b7a1c644200/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"744b38f8-98d0-4d54-be2b-8b7a1c644200\",\"--execution_id\",\"98\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"10\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/744b38f8-98d0-4d54-be2b-8b7a1c644200/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-lm5jl/2025/05/08/pod-log-to-github-lm5jl-system-container-impl-3089604882"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"744b38f8-98d0-4d54-be2b-8b7a1c644200\",\"--execution_id\",\"98\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"10\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/744b38f8-98d0-4d54-be2b-8b7a1c644200/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
              raise FileNotFoundError(f"{report_path} does not exist")
      Reason:       Error
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"744b38f8-98d0-4d54-be2b-8b7a1c644200\",\"--execution_id\",\"98\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"10\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/744b38f8-98d0-4d54-be2b-8b7a1c644200/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-lm5jl/2025/05/08/pod-log-to-github-lm5jl-system-container-impl-3089604882"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"744b38f8-98d0-4d54-be2b-8b7a1c644200\",\"--execution_id\",\"98\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"10\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/744b38f8-98d0-4d54-be2b-8b7a1c644200/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}

Logs (filtered)

remote: error: GH013: Repository rule violations found for refs/heads/main.        
error: failed to push some refs to 'https://github.com/Rosa1026/Kubeflow.git'
Traceback (most recent call last):
F0508 01:11:31.907521      22 main.go:49] failed to execute component: exit status 1
time="2025-05-08T01:11:32.898Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T01:11:33.835Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-lm5jl-system-container-impl-4245785281 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T01:10:57.008Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T01:10:57.954Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-lm5jl-system-dag-driver-155373975 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T01:09:37.622Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-lq7m2-system-container-driver-1530126834 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T00:41:05.721Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-lq7m2-system-container-driver-4246572665 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"87"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","4959d198-ddde-43ac-9b7c-ce99ae875681","--dag_execution_id","87","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-lq7m2/2025/05/08/pod-log-to-github-lq7m2-system-container-driver-4246572665"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"87"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","4959d198-ddde-43ac-9b7c-ce99ae875681","--dag_execution_id","87","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-lq7m2/2025/05/08/pod-log-to-github-lq7m2-system-container-driver-4246572665"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str = \"main\"\n):\n    import os\n    import subprocess\n\n\n    report_path = report_file.path\n    print(\"Checking if report exists at:\", report_path)\n    if not os.path.exists(report_path):\n        raise FileNotFoundError(f\"{report_path} does not exist\")\n\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    subprocess.run([\"cp\", report_path, os.path.join(repo_dir, \"report.md\")], check=True)\n\n    # Git user config\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", \"Add report\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(\"Report committed and pushed to GitHub.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"87"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","4959d198-ddde-43ac-9b7c-ce99ae875681","--dag_execution_id","87","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-lq7m2/2025/05/08/pod-log-to-github-lq7m2-system-container-driver-4246572665"}}}

Logs (filtered)

time="2025-05-08T00:42:18.524Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-lq7m2-system-container-impl-1051008407 (Namespace: kubeflow-user-example-com)

Describe (filtered)

Status:       Failed
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"4959d198-ddde-43ac-9b7c-ce99ae875681\",\"--execution_id\",\"89\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"7\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/4959d198-ddde-43ac-9b7c-ce99ae875681/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-lq7m2/2025/05/08/pod-log-to-github-lq7m2-system-container-impl-1051008407"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"4959d198-ddde-43ac-9b7c-ce99ae875681\",\"--execution_id\",\"89\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"7\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/4959d198-ddde-43ac-9b7c-ce99ae875681/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"4959d198-ddde-43ac-9b7c-ce99ae875681\",\"--execution_id\",\"89\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"7\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/4959d198-ddde-43ac-9b7c-ce99ae875681/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-lq7m2/2025/05/08/pod-log-to-github-lq7m2-system-container-impl-1051008407"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"4959d198-ddde-43ac-9b7c-ce99ae875681\",\"--execution_id\",\"89\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"7\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/4959d198-ddde-43ac-9b7c-ce99ae875681/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"4959d198-ddde-43ac-9b7c-ce99ae875681\",\"--execution_id\",\"89\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"7\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/4959d198-ddde-43ac-9b7c-ce99ae875681/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-lq7m2/2025/05/08/pod-log-to-github-lq7m2-system-container-impl-1051008407"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"4959d198-ddde-43ac-9b7c-ce99ae875681\",\"--execution_id\",\"89\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"7\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/4959d198-ddde-43ac-9b7c-ce99ae875681/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
              raise FileNotFoundError(f"{report_path} does not exist")
      Reason:       Error
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"4959d198-ddde-43ac-9b7c-ce99ae875681\",\"--execution_id\",\"89\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"7\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/4959d198-ddde-43ac-9b7c-ce99ae875681/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-lq7m2/2025/05/08/pod-log-to-github-lq7m2-system-container-impl-1051008407"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"4959d198-ddde-43ac-9b7c-ce99ae875681\",\"--execution_id\",\"89\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"7\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/4959d198-ddde-43ac-9b7c-ce99ae875681/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}

Logs (filtered)

remote: error: GH013: Repository rule violations found for refs/heads/main.        
error: failed to push some refs to 'https://github.com/Rosa1026/Kubeflow.git'
Traceback (most recent call last):
F0508 00:42:38.611503      22 main.go:49] failed to execute component: exit status 1
time="2025-05-08T00:42:39.603Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T00:42:40.538Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-lq7m2-system-container-impl-335277084 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T00:42:04.863Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T00:42:05.808Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-lq7m2-system-dag-driver-3833440696 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T00:40:55.622Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-mml78-system-container-driver-1569294087 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"113"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","29e816ea-e199-48f5-9536-cb2ab8cdc7c8","--dag_execution_id","113","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-mml78/2025/05/08/pod-log-to-github-mml78-system-container-driver-1569294087"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"113"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","29e816ea-e199-48f5-9536-cb2ab8cdc7c8","--dag_execution_id","113","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-mml78/2025/05/08/pod-log-to-github-mml78-system-container-driver-1569294087"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str = \"main\"\n):\n\n    import subprocess\n    import shutil\n    import os\n    from datetime import datetime\n\n    report_path = report_file.path\n    print(\"Checking if report exists at:\", report_path)\n    if not os.path.exists(report_path):\n        raise FileNotFoundError(f\"{report_path} does not exist\")\n\n    # HTML 파일 이름 만들기\n    timestamp = datetime.now().strftime(\"report-%Y%m%d-%H%M%S.html\")\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n    report_target_path = os.path.join(repo_dir, timestamp)\n\n    # Git clone\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    shutil.copyfile(report_path, report_target_path)\n\n    # Git config\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    # Git commit \u0026 push\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", f\"Add {timestamp}\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(f\"Report committed and pushed to GitHub as {timestamp}.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"113"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","29e816ea-e199-48f5-9536-cb2ab8cdc7c8","--dag_execution_id","113","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-mml78/2025/05/08/pod-log-to-github-mml78-system-container-driver-1569294087"}}}

Logs (filtered)

time="2025-05-08T04:56:36.223Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-mml78-system-container-driver-1787900338 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"113"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","29e816ea-e199-48f5-9536-cb2ab8cdc7c8","--dag_execution_id","113","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-mml78/2025/05/08/pod-log-to-github-mml78-system-container-driver-1787900338"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"113"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","29e816ea-e199-48f5-9536-cb2ab8cdc7c8","--dag_execution_id","113","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-mml78/2025/05/08/pod-log-to-github-mml78-system-container-driver-1787900338"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","collect_filtered_logs"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \"crash\"):\n    import shutil\n    import json\n    import subprocess\n    import os\n\n    tmp_report_path = \"/tmp/report.html\"\n    seen_namespaces = set()\n    namespace_events = {}\n\n    # kubectl 설치 방법 변경\n    subprocess.run([\"apt-get\", \"update\", \"-y\"], check=True)\n    subprocess.run([ \n        \"curl\", \"-LO\", \"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\"\n    ], check=True)\n    subprocess.run([\"chmod\", \"+x\", \"kubectl\"], check=True)\n    subprocess.run([\"mv\", \"kubectl\", \"/usr/local/bin/\"], check=True)\n\n    # Get all pods in all namespaces\n    pods_output = subprocess.getoutput(\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\")\n    pod_lines = pods_output.strip().split(\"\\n\")\n\n    with open(tmp_report_path, \"w\") as f:\n        f.write(f\"\u003chtml\u003e\u003cbody\u003e\u003ch1\u003e장애 복기 리포트 (keyword: `{keyword}`)\u003c/h1\u003e\\n\")\n\n        for line in pod_lines:\n            if not line.strip():\n                continue\n            try:\n                namespace, pod_name = line.split()\n            except ValueError:\n                continue\n\n            describe = subprocess.getoutput(f\"kubectl describe pod {pod_name} -n {namespace}\")\n            logs = subprocess.getoutput(f\"kubectl logs {pod_name} -n {namespace}\")\n\n            matched_describe = \"\\n\".join([l for l in describe.splitlines() if keyword in l])\n            matched_logs = \"\\n\".join([l for l in logs.splitlines() if keyword in l])\n\n            if matched_describe or matched_logs:\n                f.write(f\"\u003ch2\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\u003c/h2\u003e\\n\")\n                if matched_describe:\n                    f.write(f\"\u003ch3\u003eDescribe\u003c/h3\u003e\u003cpre\u003e{matched_describe}\u003c/pre\u003e\\n\")\n                if matched_logs:\n                    f.write(f\"\u003ch3\u003eLogs\u003c/h3\u003e\u003cpre\u003e{matched_logs}\u003c/pre\u003e\\n\")\n\n            if namespace not in seen_namespaces:\n                events = subprocess.getoutput(f\"kubectl get events -n {namespace}\")\n                matched_events = \"\\n\".join([l for l in events.splitlines() if keyword in l])\n                if matched_events:\n                    namespace_events[namespace] = matched_events\n                seen_namespaces.add(namespace)\n\n        # Write filtered events grouped by namespace\n        for ns, events in namespace_events.items():\n            f.write(f\"\u003ch3\u003eEvents in Namespace: `{ns}`\u003c/h3\u003e\u003cpre\u003e{events}\u003c/pre\u003e\\n\")\n\n        f.write(\"\u003c/body\u003e\u003c/html\u003e\")\n\n    subprocess.run([\"cp\", tmp_report_path, output_report.path], check=True)\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"113"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","29e816ea-e199-48f5-9536-cb2ab8cdc7c8","--dag_execution_id","113","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-mml78/2025/05/08/pod-log-to-github-mml78-system-container-driver-1787900338"}}}

Logs (filtered)

time="2025-05-08T04:55:05.222Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-mml78-system-container-impl-3955645660 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29e816ea-e199-48f5-9536-cb2ab8cdc7c8\",\"--execution_id\",\"114\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29e816ea-e199-48f5-9536-cb2ab8cdc7c8/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-mml78/2025/05/08/pod-log-to-github-mml78-system-container-impl-3955645660"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29e816ea-e199-48f5-9536-cb2ab8cdc7c8\",\"--execution_id\",\"114\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29e816ea-e199-48f5-9536-cb2ab8cdc7c8/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29e816ea-e199-48f5-9536-cb2ab8cdc7c8\",\"--execution_id\",\"114\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29e816ea-e199-48f5-9536-cb2ab8cdc7c8/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-mml78/2025/05/08/pod-log-to-github-mml78-system-container-impl-3955645660"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29e816ea-e199-48f5-9536-cb2ab8cdc7c8\",\"--execution_id\",\"114\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29e816ea-e199-48f5-9536-cb2ab8cdc7c8/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29e816ea-e199-48f5-9536-cb2ab8cdc7c8\",\"--execution_id\",\"114\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29e816ea-e199-48f5-9536-cb2ab8cdc7c8/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-mml78/2025/05/08/pod-log-to-github-mml78-system-container-impl-3955645660"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29e816ea-e199-48f5-9536-cb2ab8cdc7c8\",\"--execution_id\",\"114\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29e816ea-e199-48f5-9536-cb2ab8cdc7c8/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
                  except ValueError:
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29e816ea-e199-48f5-9536-cb2ab8cdc7c8\",\"--execution_id\",\"114\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29e816ea-e199-48f5-9536-cb2ab8cdc7c8/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-mml78/2025/05/08/pod-log-to-github-mml78-system-container-impl-3955645660"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29e816ea-e199-48f5-9536-cb2ab8cdc7c8\",\"--execution_id\",\"114\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29e816ea-e199-48f5-9536-cb2ab8cdc7c8/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}

Logs (filtered)

time="2025-05-08T04:56:22.398Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T04:56:23.347Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-mml78-system-container-impl-4210283497 (Namespace: kubeflow-user-example-com)

Describe (filtered)

Status:       Failed
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29e816ea-e199-48f5-9536-cb2ab8cdc7c8\",\"--execution_id\",\"115\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"15\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29e816ea-e199-48f5-9536-cb2ab8cdc7c8/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-mml78/2025/05/08/pod-log-to-github-mml78-system-container-impl-4210283497"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29e816ea-e199-48f5-9536-cb2ab8cdc7c8\",\"--execution_id\",\"115\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"15\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29e816ea-e199-48f5-9536-cb2ab8cdc7c8/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29e816ea-e199-48f5-9536-cb2ab8cdc7c8\",\"--execution_id\",\"115\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"15\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29e816ea-e199-48f5-9536-cb2ab8cdc7c8/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-mml78/2025/05/08/pod-log-to-github-mml78-system-container-impl-4210283497"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29e816ea-e199-48f5-9536-cb2ab8cdc7c8\",\"--execution_id\",\"115\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"15\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29e816ea-e199-48f5-9536-cb2ab8cdc7c8/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29e816ea-e199-48f5-9536-cb2ab8cdc7c8\",\"--execution_id\",\"115\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"15\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29e816ea-e199-48f5-9536-cb2ab8cdc7c8/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-mml78/2025/05/08/pod-log-to-github-mml78-system-container-impl-4210283497"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29e816ea-e199-48f5-9536-cb2ab8cdc7c8\",\"--execution_id\",\"115\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"15\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29e816ea-e199-48f5-9536-cb2ab8cdc7c8/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
              raise FileNotFoundError(f"{report_path} does not exist")
      Reason:       Error
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29e816ea-e199-48f5-9536-cb2ab8cdc7c8\",\"--execution_id\",\"115\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"15\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29e816ea-e199-48f5-9536-cb2ab8cdc7c8/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-mml78/2025/05/08/pod-log-to-github-mml78-system-container-impl-4210283497"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29e816ea-e199-48f5-9536-cb2ab8cdc7c8\",\"--execution_id\",\"115\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"15\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29e816ea-e199-48f5-9536-cb2ab8cdc7c8/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}

Logs (filtered)

Traceback (most recent call last):
F0508 04:56:55.066794      24 main.go:49] failed to execute component: exit status 1
time="2025-05-08T04:56:55.462Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T04:56:56.403Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-mml78-system-dag-driver-4015478838 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T04:54:55.031Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-nm76f-system-container-driver-2488320014 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"140"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","065e0dae-405c-48d9-be83-a747d80b8ae0","--dag_execution_id","140","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-nm76f/2025/05/08/pod-log-to-github-nm76f-system-container-driver-2488320014"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"140"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","065e0dae-405c-48d9-be83-a747d80b8ae0","--dag_execution_id","140","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-nm76f/2025/05/08/pod-log-to-github-nm76f-system-container-driver-2488320014"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str\n):\n\n    import subprocess\n    import shutil\n    import os\n    from datetime import datetime\n\n    os.environ[\"GIT_ASKPASS\"] = \"echo\"\n\n    report_path = report_file.path\n    print(\"Checking if report exists at:\", report_path)\n    if not os.path.exists(report_path):\n        raise FileNotFoundError(f\"{report_path} does not exist\")\n\n    # HTML 파일 이름 만들기\n    timestamp = datetime.now().strftime(\"report-%Y%m%d-%H%M%S.html\")\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n    report_target_path = os.path.join(repo_dir, timestamp)\n\n    # Git clone\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    shutil.copyfile(report_path, report_target_path)\n\n    # Git config\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    # Git commit \u0026 push\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", f\"Add {timestamp}\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(f\"Report committed and pushed to GitHub as {timestamp}.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"140"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","065e0dae-405c-48d9-be83-a747d80b8ae0","--dag_execution_id","140","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-nm76f/2025/05/08/pod-log-to-github-nm76f-system-container-driver-2488320014"}}}

Logs (filtered)

time="2025-05-08T06:14:34.523Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-nm76f-system-container-driver-3397474225 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n    report_data = []\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # 모든 Pod 가져오기\\n    pods_output = subprocess.getoutput(\\n        \\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\"\\n    )\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    for line in pod_lines:\\n        if not line.strip():\\n            continue\\n        try:\\n            namespace, pod_name = line.split()\\n        except ValueError:\\n            continue\\n\\n        describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n        logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n        matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n        matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n        if matched_describe or matched_logs:\\n            report_data.append({\\n                \\\"namespace\\\": namespace,\\n                \\\"pod\\\": pod_name,\\n                \\\"describe\\\": matched_describe,\\n                \\\"logs\\\": matched_logs\\n            })\\n\\n        if namespace not in seen_namespaces:\\n            events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n            matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n            if matched_events:\\n                namespace_events[namespace] = matched_events\\n            seen_namespaces.add(namespace)\\n\\n    # HTML 작성\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(\\\"\\\"\\\"\\n\\u003c!DOCTYPE html\\u003e\\n\\u003chtml lang=\\\"en\\\"\\u003e\\n\\u003chead\\u003e\\n    \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n    \\u003ctitle\\u003eKubeflow Log Report\\u003c/title\\u003e\\n    \\u003cstyle\\u003e\\n        body {\\n            font-family: 'Segoe UI', sans-serif;\\n            background-color: #f9f9f9;\\n            color: #333;\\n            padding: 2em;\\n            line-height: 1.6;\\n        }\\n        h1 {\\n            color: #004080;\\n            text-align: center;\\n        }\\n        .namespace {\\n            margin-bottom: 2em;\\n            border: 1px solid #ccc;\\n            padding: 1em;\\n            background-color: #fff;\\n            box-shadow: 0 2px 5px rgba(0,0,0,0.1);\\n            border-radius: 8px;\\n        }\\n        .pod-name {\\n            font-weight: bold;\\n            margin-top: 1em;\\n            font-size: 1.2em;\\n            color: #2a5885;\\n        }\\n        .section-title {\\n            margin-top: 1em;\\n            color: #555;\\n            font-size: 1em;\\n            border-bottom: 1px solid #ddd;\\n        }\\n        pre {\\n            background-color: #272822;\\n            color: #f8f8f2;\\n            padding: 1em;\\n            overflow-x: auto;\\n            white-space: pre-wrap;\\n            border-radius: 4px;\\n        }\\n        .events {\\n            background-color: #fffaf0;\\n            border-left: 5px solid #ffa500;\\n            padding: 1em;\\n            margin-top: 1em;\\n        }\\n    \\u003c/style\\u003e\\n\\u003c/head\\u003e\\n\\u003cbody\\u003e\\n\\u003ch1\\u003eKubeflow Pod Diagnostics Report\\u003c/h1\\u003e\\n\\\"\\\"\\\")\\n\\n        for entry in report_data:\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003ePod: {entry[\\\"pod\\\"]} (Namespace: {entry[\\\"namespace\\\"]})\\u003c/div\\u003e\\\\n')\\n            if entry.get(\\\"describe\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eDescribe\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"describe\\\"]}\\u003c/pre\\u003e\\\\n')\\n            if entry.get(\\\"logs\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eLogs\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"logs\\\"]}\\u003c/pre\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003eEvents in Namespace: {ns}\\u003c/div\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"events\\\"\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/div\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"140"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","065e0dae-405c-48d9-be83-a747d80b8ae0","--dag_execution_id","140","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n    report_data = []\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # 모든 Pod 가져오기\\n    pods_output = subprocess.getoutput(\\n        \\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\"\\n    )\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    for line in pod_lines:\\n        if not line.strip():\\n            continue\\n        try:\\n            namespace, pod_name = line.split()\\n        except ValueError:\\n            continue\\n\\n        describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n        logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n        matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n        matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n        if matched_describe or matched_logs:\\n            report_data.append({\\n                \\\"namespace\\\": namespace,\\n                \\\"pod\\\": pod_name,\\n                \\\"describe\\\": matched_describe,\\n                \\\"logs\\\": matched_logs\\n            })\\n\\n        if namespace not in seen_namespaces:\\n            events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n            matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n            if matched_events:\\n                namespace_events[namespace] = matched_events\\n            seen_namespaces.add(namespace)\\n\\n    # HTML 작성\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(\\\"\\\"\\\"\\n\\u003c!DOCTYPE html\\u003e\\n\\u003chtml lang=\\\"en\\\"\\u003e\\n\\u003chead\\u003e\\n    \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n    \\u003ctitle\\u003eKubeflow Log Report\\u003c/title\\u003e\\n    \\u003cstyle\\u003e\\n        body {\\n            font-family: 'Segoe UI', sans-serif;\\n            background-color: #f9f9f9;\\n            color: #333;\\n            padding: 2em;\\n            line-height: 1.6;\\n        }\\n        h1 {\\n            color: #004080;\\n            text-align: center;\\n        }\\n        .namespace {\\n            margin-bottom: 2em;\\n            border: 1px solid #ccc;\\n            padding: 1em;\\n            background-color: #fff;\\n            box-shadow: 0 2px 5px rgba(0,0,0,0.1);\\n            border-radius: 8px;\\n        }\\n        .pod-name {\\n            font-weight: bold;\\n            margin-top: 1em;\\n            font-size: 1.2em;\\n            color: #2a5885;\\n        }\\n        .section-title {\\n            margin-top: 1em;\\n            color: #555;\\n            font-size: 1em;\\n            border-bottom: 1px solid #ddd;\\n        }\\n        pre {\\n            background-color: #272822;\\n            color: #f8f8f2;\\n            padding: 1em;\\n            overflow-x: auto;\\n            white-space: pre-wrap;\\n            border-radius: 4px;\\n        }\\n        .events {\\n            background-color: #fffaf0;\\n            border-left: 5px solid #ffa500;\\n            padding: 1em;\\n            margin-top: 1em;\\n        }\\n    \\u003c/style\\u003e\\n\\u003c/head\\u003e\\n\\u003cbody\\u003e\\n\\u003ch1\\u003eKubeflow Pod Diagnostics Report\\u003c/h1\\u003e\\n\\\"\\\"\\\")\\n\\n        for entry in report_data:\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003ePod: {entry[\\\"pod\\\"]} (Namespace: {entry[\\\"namespace\\\"]})\\u003c/div\\u003e\\\\n')\\n            if entry.get(\\\"describe\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eDescribe\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"describe\\\"]}\\u003c/pre\\u003e\\\\n')\\n            if entry.get(\\\"logs\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eLogs\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"logs\\\"]}\\u003c/pre\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003eEvents in Namespace: {ns}\\u003c/div\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"events\\\"\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/div\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-nm76f/2025/05/08/pod-log-to-github-nm76f-system-container-driver-3397474225"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n    report_data = []\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # 모든 Pod 가져오기\\n    pods_output = subprocess.getoutput(\\n        \\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\"\\n    )\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    for line in pod_lines:\\n        if not line.strip():\\n            continue\\n        try:\\n            namespace, pod_name = line.split()\\n        except ValueError:\\n            continue\\n\\n        describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n        logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n        matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n        matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n        if matched_describe or matched_logs:\\n            report_data.append({\\n                \\\"namespace\\\": namespace,\\n                \\\"pod\\\": pod_name,\\n                \\\"describe\\\": matched_describe,\\n                \\\"logs\\\": matched_logs\\n            })\\n\\n        if namespace not in seen_namespaces:\\n            events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n            matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n            if matched_events:\\n                namespace_events[namespace] = matched_events\\n            seen_namespaces.add(namespace)\\n\\n    # HTML 작성\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(\\\"\\\"\\\"\\n\\u003c!DOCTYPE html\\u003e\\n\\u003chtml lang=\\\"en\\\"\\u003e\\n\\u003chead\\u003e\\n    \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n    \\u003ctitle\\u003eKubeflow Log Report\\u003c/title\\u003e\\n    \\u003cstyle\\u003e\\n        body {\\n            font-family: 'Segoe UI', sans-serif;\\n            background-color: #f9f9f9;\\n            color: #333;\\n            padding: 2em;\\n            line-height: 1.6;\\n        }\\n        h1 {\\n            color: #004080;\\n            text-align: center;\\n        }\\n        .namespace {\\n            margin-bottom: 2em;\\n            border: 1px solid #ccc;\\n            padding: 1em;\\n            background-color: #fff;\\n            box-shadow: 0 2px 5px rgba(0,0,0,0.1);\\n            border-radius: 8px;\\n        }\\n        .pod-name {\\n            font-weight: bold;\\n            margin-top: 1em;\\n            font-size: 1.2em;\\n            color: #2a5885;\\n        }\\n        .section-title {\\n            margin-top: 1em;\\n            color: #555;\\n            font-size: 1em;\\n            border-bottom: 1px solid #ddd;\\n        }\\n        pre {\\n            background-color: #272822;\\n            color: #f8f8f2;\\n            padding: 1em;\\n            overflow-x: auto;\\n            white-space: pre-wrap;\\n            border-radius: 4px;\\n        }\\n        .events {\\n            background-color: #fffaf0;\\n            border-left: 5px solid #ffa500;\\n            padding: 1em;\\n            margin-top: 1em;\\n        }\\n    \\u003c/style\\u003e\\n\\u003c/head\\u003e\\n\\u003cbody\\u003e\\n\\u003ch1\\u003eKubeflow Pod Diagnostics Report\\u003c/h1\\u003e\\n\\\"\\\"\\\")\\n\\n        for entry in report_data:\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003ePod: {entry[\\\"pod\\\"]} (Namespace: {entry[\\\"namespace\\\"]})\\u003c/div\\u003e\\\\n')\\n            if entry.get(\\\"describe\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eDescribe\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"describe\\\"]}\\u003c/pre\\u003e\\\\n')\\n            if entry.get(\\\"logs\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eLogs\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"logs\\\"]}\\u003c/pre\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003eEvents in Namespace: {ns}\\u003c/div\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"events\\\"\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/div\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"140"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","065e0dae-405c-48d9-be83-a747d80b8ae0","--dag_execution_id","140","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n    report_data = []\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # 모든 Pod 가져오기\\n    pods_output = subprocess.getoutput(\\n        \\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\"\\n    )\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    for line in pod_lines:\\n        if not line.strip():\\n            continue\\n        try:\\n            namespace, pod_name = line.split()\\n        except ValueError:\\n            continue\\n\\n        describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n        logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n        matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n        matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n        if matched_describe or matched_logs:\\n            report_data.append({\\n                \\\"namespace\\\": namespace,\\n                \\\"pod\\\": pod_name,\\n                \\\"describe\\\": matched_describe,\\n                \\\"logs\\\": matched_logs\\n            })\\n\\n        if namespace not in seen_namespaces:\\n            events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n            matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n            if matched_events:\\n                namespace_events[namespace] = matched_events\\n            seen_namespaces.add(namespace)\\n\\n    # HTML 작성\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(\\\"\\\"\\\"\\n\\u003c!DOCTYPE html\\u003e\\n\\u003chtml lang=\\\"en\\\"\\u003e\\n\\u003chead\\u003e\\n    \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n    \\u003ctitle\\u003eKubeflow Log Report\\u003c/title\\u003e\\n    \\u003cstyle\\u003e\\n        body {\\n            font-family: 'Segoe UI', sans-serif;\\n            background-color: #f9f9f9;\\n            color: #333;\\n            padding: 2em;\\n            line-height: 1.6;\\n        }\\n        h1 {\\n            color: #004080;\\n            text-align: center;\\n        }\\n        .namespace {\\n            margin-bottom: 2em;\\n            border: 1px solid #ccc;\\n            padding: 1em;\\n            background-color: #fff;\\n            box-shadow: 0 2px 5px rgba(0,0,0,0.1);\\n            border-radius: 8px;\\n        }\\n        .pod-name {\\n            font-weight: bold;\\n            margin-top: 1em;\\n            font-size: 1.2em;\\n            color: #2a5885;\\n        }\\n        .section-title {\\n            margin-top: 1em;\\n            color: #555;\\n            font-size: 1em;\\n            border-bottom: 1px solid #ddd;\\n        }\\n        pre {\\n            background-color: #272822;\\n            color: #f8f8f2;\\n            padding: 1em;\\n            overflow-x: auto;\\n            white-space: pre-wrap;\\n            border-radius: 4px;\\n        }\\n        .events {\\n            background-color: #fffaf0;\\n            border-left: 5px solid #ffa500;\\n            padding: 1em;\\n            margin-top: 1em;\\n        }\\n    \\u003c/style\\u003e\\n\\u003c/head\\u003e\\n\\u003cbody\\u003e\\n\\u003ch1\\u003eKubeflow Pod Diagnostics Report\\u003c/h1\\u003e\\n\\\"\\\"\\\")\\n\\n        for entry in report_data:\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003ePod: {entry[\\\"pod\\\"]} (Namespace: {entry[\\\"namespace\\\"]})\\u003c/div\\u003e\\\\n')\\n            if entry.get(\\\"describe\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eDescribe\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"describe\\\"]}\\u003c/pre\\u003e\\\\n')\\n            if entry.get(\\\"logs\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eLogs\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"logs\\\"]}\\u003c/pre\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003eEvents in Namespace: {ns}\\u003c/div\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"events\\\"\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/div\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-nm76f/2025/05/08/pod-log-to-github-nm76f-system-container-driver-3397474225"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","collect_filtered_logs"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \"crash\"):\n    import shutil\n    import subprocess\n    import os\n\n    tmp_report_path = \"/tmp/report.html\"\n    seen_namespaces = set()\n    namespace_events = {}\n    report_data = []\n\n    # kubectl 설치\n    subprocess.run([\"apt-get\", \"update\", \"-y\"], check=True)\n    subprocess.run([\n        \"curl\", \"-LO\", \"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\"\n    ], check=True)\n    subprocess.run([\"chmod\", \"+x\", \"kubectl\"], check=True)\n    subprocess.run([\"mv\", \"kubectl\", \"/usr/local/bin/\"], check=True)\n\n    # 모든 Pod 가져오기\n    pods_output = subprocess.getoutput(\n        \"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\"\n    )\n    pod_lines = pods_output.strip().split(\"\\n\")\n\n    for line in pod_lines:\n        if not line.strip():\n            continue\n        try:\n            namespace, pod_name = line.split()\n        except ValueError:\n            continue\n\n        describe = subprocess.getoutput(f\"kubectl describe pod {pod_name} -n {namespace}\")\n        logs = subprocess.getoutput(f\"kubectl logs {pod_name} -n {namespace}\")\n\n        matched_describe = \"\\n\".join([l for l in describe.splitlines() if keyword in l])\n        matched_logs = \"\\n\".join([l for l in logs.splitlines() if keyword in l])\n\n        if matched_describe or matched_logs:\n            report_data.append({\n                \"namespace\": namespace,\n                \"pod\": pod_name,\n                \"describe\": matched_describe,\n                \"logs\": matched_logs\n            })\n\n        if namespace not in seen_namespaces:\n            events = subprocess.getoutput(f\"kubectl get events -n {namespace}\")\n            matched_events = \"\\n\".join([l for l in events.splitlines() if keyword in l])\n            if matched_events:\n                namespace_events[namespace] = matched_events\n            seen_namespaces.add(namespace)\n\n    # HTML 작성\n    with open(tmp_report_path, \"w\") as f:\n        f.write(\"\"\"\n\u003c!DOCTYPE html\u003e\n\u003chtml lang=\"en\"\u003e\n\u003chead\u003e\n    \u003cmeta charset=\"UTF-8\"\u003e\n    \u003ctitle\u003eKubeflow Log Report\u003c/title\u003e\n    \u003cstyle\u003e\n        body {\n            font-family: 'Segoe UI', sans-serif;\n            background-color: #f9f9f9;\n            color: #333;\n            padding: 2em;\n            line-height: 1.6;\n        }\n        h1 {\n            color: #004080;\n            text-align: center;\n        }\n        .namespace {\n            margin-bottom: 2em;\n            border: 1px solid #ccc;\n            padding: 1em;\n            background-color: #fff;\n            box-shadow: 0 2px 5px rgba(0,0,0,0.1);\n            border-radius: 8px;\n        }\n        .pod-name {\n            font-weight: bold;\n            margin-top: 1em;\n            font-size: 1.2em;\n            color: #2a5885;\n        }\n        .section-title {\n            margin-top: 1em;\n            color: #555;\n            font-size: 1em;\n            border-bottom: 1px solid #ddd;\n        }\n        pre {\n            background-color: #272822;\n            color: #f8f8f2;\n            padding: 1em;\n            overflow-x: auto;\n            white-space: pre-wrap;\n            border-radius: 4px;\n        }\n        .events {\n            background-color: #fffaf0;\n            border-left: 5px solid #ffa500;\n            padding: 1em;\n            margin-top: 1em;\n        }\n    \u003c/style\u003e\n\u003c/head\u003e\n\u003cbody\u003e\n\u003ch1\u003eKubeflow Pod Diagnostics Report\u003c/h1\u003e\n\"\"\")\n\n        for entry in report_data:\n            f.write(f'\u003cdiv class=\"namespace\"\u003e\\n')\n            f.write(f'\u003cdiv class=\"pod-name\"\u003ePod: {entry[\"pod\"]} (Namespace: {entry[\"namespace\"]})\u003c/div\u003e\\n')\n            if entry.get(\"describe\"):\n                f.write(f'\u003cdiv class=\"section-title\"\u003eDescribe\u003c/div\u003e\\n\u003cpre\u003e{entry[\"describe\"]}\u003c/pre\u003e\\n')\n            if entry.get(\"logs\"):\n                f.write(f'\u003cdiv class=\"section-title\"\u003eLogs\u003c/div\u003e\\n\u003cpre\u003e{entry[\"logs\"]}\u003c/pre\u003e\\n')\n            f.write('\u003c/div\u003e\\n')\n\n        for ns, events in namespace_events.items():\n            f.write(f'\u003cdiv class=\"namespace\"\u003e\\n')\n            f.write(f'\u003cdiv class=\"pod-name\"\u003eEvents in Namespace: {ns}\u003c/div\u003e\\n')\n            f.write(f'\u003cdiv class=\"events\"\u003e\u003cpre\u003e{events}\u003c/pre\u003e\u003c/div\u003e\\n')\n            f.write('\u003c/div\u003e\\n')\n\n        f.write(\"\u003c/body\u003e\u003c/html\u003e\")\n\n    subprocess.run([\"cp\", tmp_report_path, output_report.path], check=True)\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n    report_data = []\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # 모든 Pod 가져오기\\n    pods_output = subprocess.getoutput(\\n        \\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\"\\n    )\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    for line in pod_lines:\\n        if not line.strip():\\n            continue\\n        try:\\n            namespace, pod_name = line.split()\\n        except ValueError:\\n            continue\\n\\n        describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n        logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n        matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n        matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n        if matched_describe or matched_logs:\\n            report_data.append({\\n                \\\"namespace\\\": namespace,\\n                \\\"pod\\\": pod_name,\\n                \\\"describe\\\": matched_describe,\\n                \\\"logs\\\": matched_logs\\n            })\\n\\n        if namespace not in seen_namespaces:\\n            events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n            matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n            if matched_events:\\n                namespace_events[namespace] = matched_events\\n            seen_namespaces.add(namespace)\\n\\n    # HTML 작성\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(\\\"\\\"\\\"\\n\\u003c!DOCTYPE html\\u003e\\n\\u003chtml lang=\\\"en\\\"\\u003e\\n\\u003chead\\u003e\\n    \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n    \\u003ctitle\\u003eKubeflow Log Report\\u003c/title\\u003e\\n    \\u003cstyle\\u003e\\n        body {\\n            font-family: 'Segoe UI', sans-serif;\\n            background-color: #f9f9f9;\\n            color: #333;\\n            padding: 2em;\\n            line-height: 1.6;\\n        }\\n        h1 {\\n            color: #004080;\\n            text-align: center;\\n        }\\n        .namespace {\\n            margin-bottom: 2em;\\n            border: 1px solid #ccc;\\n            padding: 1em;\\n            background-color: #fff;\\n            box-shadow: 0 2px 5px rgba(0,0,0,0.1);\\n            border-radius: 8px;\\n        }\\n        .pod-name {\\n            font-weight: bold;\\n            margin-top: 1em;\\n            font-size: 1.2em;\\n            color: #2a5885;\\n        }\\n        .section-title {\\n            margin-top: 1em;\\n            color: #555;\\n            font-size: 1em;\\n            border-bottom: 1px solid #ddd;\\n        }\\n        pre {\\n            background-color: #272822;\\n            color: #f8f8f2;\\n            padding: 1em;\\n            overflow-x: auto;\\n            white-space: pre-wrap;\\n            border-radius: 4px;\\n        }\\n        .events {\\n            background-color: #fffaf0;\\n            border-left: 5px solid #ffa500;\\n            padding: 1em;\\n            margin-top: 1em;\\n        }\\n    \\u003c/style\\u003e\\n\\u003c/head\\u003e\\n\\u003cbody\\u003e\\n\\u003ch1\\u003eKubeflow Pod Diagnostics Report\\u003c/h1\\u003e\\n\\\"\\\"\\\")\\n\\n        for entry in report_data:\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003ePod: {entry[\\\"pod\\\"]} (Namespace: {entry[\\\"namespace\\\"]})\\u003c/div\\u003e\\\\n')\\n            if entry.get(\\\"describe\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eDescribe\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"describe\\\"]}\\u003c/pre\\u003e\\\\n')\\n            if entry.get(\\\"logs\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eLogs\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"logs\\\"]}\\u003c/pre\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003eEvents in Namespace: {ns}\\u003c/div\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"events\\\"\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/div\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"140"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","065e0dae-405c-48d9-be83-a747d80b8ae0","--dag_execution_id","140","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n    report_data = []\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # 모든 Pod 가져오기\\n    pods_output = subprocess.getoutput(\\n        \\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\"\\n    )\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    for line in pod_lines:\\n        if not line.strip():\\n            continue\\n        try:\\n            namespace, pod_name = line.split()\\n        except ValueError:\\n            continue\\n\\n        describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n        logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n        matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n        matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n        if matched_describe or matched_logs:\\n            report_data.append({\\n                \\\"namespace\\\": namespace,\\n                \\\"pod\\\": pod_name,\\n                \\\"describe\\\": matched_describe,\\n                \\\"logs\\\": matched_logs\\n            })\\n\\n        if namespace not in seen_namespaces:\\n            events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n            matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n            if matched_events:\\n                namespace_events[namespace] = matched_events\\n            seen_namespaces.add(namespace)\\n\\n    # HTML 작성\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(\\\"\\\"\\\"\\n\\u003c!DOCTYPE html\\u003e\\n\\u003chtml lang=\\\"en\\\"\\u003e\\n\\u003chead\\u003e\\n    \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n    \\u003ctitle\\u003eKubeflow Log Report\\u003c/title\\u003e\\n    \\u003cstyle\\u003e\\n        body {\\n            font-family: 'Segoe UI', sans-serif;\\n            background-color: #f9f9f9;\\n            color: #333;\\n            padding: 2em;\\n            line-height: 1.6;\\n        }\\n        h1 {\\n            color: #004080;\\n            text-align: center;\\n        }\\n        .namespace {\\n            margin-bottom: 2em;\\n            border: 1px solid #ccc;\\n            padding: 1em;\\n            background-color: #fff;\\n            box-shadow: 0 2px 5px rgba(0,0,0,0.1);\\n            border-radius: 8px;\\n        }\\n        .pod-name {\\n            font-weight: bold;\\n            margin-top: 1em;\\n            font-size: 1.2em;\\n            color: #2a5885;\\n        }\\n        .section-title {\\n            margin-top: 1em;\\n            color: #555;\\n            font-size: 1em;\\n            border-bottom: 1px solid #ddd;\\n        }\\n        pre {\\n            background-color: #272822;\\n            color: #f8f8f2;\\n            padding: 1em;\\n            overflow-x: auto;\\n            white-space: pre-wrap;\\n            border-radius: 4px;\\n        }\\n        .events {\\n            background-color: #fffaf0;\\n            border-left: 5px solid #ffa500;\\n            padding: 1em;\\n            margin-top: 1em;\\n        }\\n    \\u003c/style\\u003e\\n\\u003c/head\\u003e\\n\\u003cbody\\u003e\\n\\u003ch1\\u003eKubeflow Pod Diagnostics Report\\u003c/h1\\u003e\\n\\\"\\\"\\\")\\n\\n        for entry in report_data:\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003ePod: {entry[\\\"pod\\\"]} (Namespace: {entry[\\\"namespace\\\"]})\\u003c/div\\u003e\\\\n')\\n            if entry.get(\\\"describe\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eDescribe\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"describe\\\"]}\\u003c/pre\\u003e\\\\n')\\n            if entry.get(\\\"logs\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eLogs\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"logs\\\"]}\\u003c/pre\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003eEvents in Namespace: {ns}\\u003c/div\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"events\\\"\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/div\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-nm76f/2025/05/08/pod-log-to-github-nm76f-system-container-driver-3397474225"}}}

Logs (filtered)

time="2025-05-08T06:12:59.318Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-nm76f-system-container-impl-3124929480 (Namespace: kubeflow-user-example-com)

Describe (filtered)

Status:       Failed
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"065e0dae-405c-48d9-be83-a747d80b8ae0\",\"--execution_id\",\"142\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"24\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/065e0dae-405c-48d9-be83-a747d80b8ae0/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-nm76f/2025/05/08/pod-log-to-github-nm76f-system-container-impl-3124929480"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"065e0dae-405c-48d9-be83-a747d80b8ae0\",\"--execution_id\",\"142\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"24\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/065e0dae-405c-48d9-be83-a747d80b8ae0/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"065e0dae-405c-48d9-be83-a747d80b8ae0\",\"--execution_id\",\"142\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"24\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/065e0dae-405c-48d9-be83-a747d80b8ae0/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-nm76f/2025/05/08/pod-log-to-github-nm76f-system-container-impl-3124929480"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"065e0dae-405c-48d9-be83-a747d80b8ae0\",\"--execution_id\",\"142\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"24\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/065e0dae-405c-48d9-be83-a747d80b8ae0/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"065e0dae-405c-48d9-be83-a747d80b8ae0\",\"--execution_id\",\"142\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"24\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/065e0dae-405c-48d9-be83-a747d80b8ae0/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-nm76f/2025/05/08/pod-log-to-github-nm76f-system-container-impl-3124929480"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"065e0dae-405c-48d9-be83-a747d80b8ae0\",\"--execution_id\",\"142\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"24\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/065e0dae-405c-48d9-be83-a747d80b8ae0/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
              raise FileNotFoundError(f"{report_path} does not exist")
      Reason:       Error
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"065e0dae-405c-48d9-be83-a747d80b8ae0\",\"--execution_id\",\"142\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"24\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/065e0dae-405c-48d9-be83-a747d80b8ae0/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-nm76f/2025/05/08/pod-log-to-github-nm76f-system-container-impl-3124929480"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"065e0dae-405c-48d9-be83-a747d80b8ae0\",\"--execution_id\",\"142\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"24\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/065e0dae-405c-48d9-be83-a747d80b8ae0/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}

Logs (filtered)

fatal: Authentication failed for 'https://github.com/Rosa1026/Kubeflow.git/'
Traceback (most recent call last):
F0508 06:14:53.794481      22 main.go:49] failed to execute component: exit status 1
time="2025-05-08T06:14:53.813Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T06:14:54.751Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-nm76f-system-container-impl-518703743 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"065e0dae-405c-48d9-be83-a747d80b8ae0\",\"--execution_id\",\"141\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/065e0dae-405c-48d9-be83-a747d80b8ae0/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n    report_data = []\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # 모든 Pod 가져오기\\n    pods_output = subprocess.getoutput(\\n        \\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\"\\n    )\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    for line in pod_lines:\\n        if not line.strip():\\n            continue\\n        try:\\n            namespace, pod_name = line.split()\\n        except ValueError:\\n            continue\\n\\n        describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n        logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n        matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n        matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n        if matched_describe or matched_logs:\\n            report_data.append({\\n                \\\"namespace\\\": namespace,\\n                \\\"pod\\\": pod_name,\\n                \\\"describe\\\": matched_describe,\\n                \\\"logs\\\": matched_logs\\n            })\\n\\n        if namespace not in seen_namespaces:\\n            events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n            matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n            if matched_events:\\n                namespace_events[namespace] = matched_events\\n            seen_namespaces.add(namespace)\\n\\n    # HTML 작성\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(\\\"\\\"\\\"\\n\\u003c!DOCTYPE html\\u003e\\n\\u003chtml lang=\\\"en\\\"\\u003e\\n\\u003chead\\u003e\\n    \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n    \\u003ctitle\\u003eKubeflow Log Report\\u003c/title\\u003e\\n    \\u003cstyle\\u003e\\n        body {\\n            font-family: 'Segoe UI', sans-serif;\\n            background-color: #f9f9f9;\\n            color: #333;\\n            padding: 2em;\\n            line-height: 1.6;\\n        }\\n        h1 {\\n            color: #004080;\\n            text-align: center;\\n        }\\n        .namespace {\\n            margin-bottom: 2em;\\n            border: 1px solid #ccc;\\n            padding: 1em;\\n            background-color: #fff;\\n            box-shadow: 0 2px 5px rgba(0,0,0,0.1);\\n            border-radius: 8px;\\n        }\\n        .pod-name {\\n            font-weight: bold;\\n            margin-top: 1em;\\n            font-size: 1.2em;\\n            color: #2a5885;\\n        }\\n        .section-title {\\n            margin-top: 1em;\\n            color: #555;\\n            font-size: 1em;\\n            border-bottom: 1px solid #ddd;\\n        }\\n        pre {\\n            background-color: #272822;\\n            color: #f8f8f2;\\n            padding: 1em;\\n            overflow-x: auto;\\n            white-space: pre-wrap;\\n            border-radius: 4px;\\n        }\\n        .events {\\n            background-color: #fffaf0;\\n            border-left: 5px solid #ffa500;\\n            padding: 1em;\\n            margin-top: 1em;\\n        }\\n    \\u003c/style\\u003e\\n\\u003c/head\\u003e\\n\\u003cbody\\u003e\\n\\u003ch1\\u003eKubeflow Pod Diagnostics Report\\u003c/h1\\u003e\\n\\\"\\\"\\\")\\n\\n        for entry in report_data:\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003ePod: {entry[\\\"pod\\\"]} (Namespace: {entry[\\\"namespace\\\"]})\\u003c/div\\u003e\\\\n')\\n            if entry.get(\\\"describe\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eDescribe\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"describe\\\"]}\\u003c/pre\\u003e\\\\n')\\n            if entry.get(\\\"logs\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eLogs\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"logs\\\"]}\\u003c/pre\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003eEvents in Namespace: {ns}\\u003c/div\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"events\\\"\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/div\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-nm76f/2025/05/08/pod-log-to-github-nm76f-system-container-impl-518703743"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"065e0dae-405c-48d9-be83-a747d80b8ae0\",\"--execution_id\",\"141\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/065e0dae-405c-48d9-be83-a747d80b8ae0/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n    report_data = []\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # 모든 Pod 가져오기\\n    pods_output = subprocess.getoutput(\\n        \\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\"\\n    )\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    for line in pod_lines:\\n        if not line.strip():\\n            continue\\n        try:\\n            namespace, pod_name = line.split()\\n        except ValueError:\\n            continue\\n\\n        describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n        logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n        matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n        matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n        if matched_describe or matched_logs:\\n            report_data.append({\\n                \\\"namespace\\\": namespace,\\n                \\\"pod\\\": pod_name,\\n                \\\"describe\\\": matched_describe,\\n                \\\"logs\\\": matched_logs\\n            })\\n\\n        if namespace not in seen_namespaces:\\n            events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n            matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n            if matched_events:\\n                namespace_events[namespace] = matched_events\\n            seen_namespaces.add(namespace)\\n\\n    # HTML 작성\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(\\\"\\\"\\\"\\n\\u003c!DOCTYPE html\\u003e\\n\\u003chtml lang=\\\"en\\\"\\u003e\\n\\u003chead\\u003e\\n    \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n    \\u003ctitle\\u003eKubeflow Log Report\\u003c/title\\u003e\\n    \\u003cstyle\\u003e\\n        body {\\n            font-family: 'Segoe UI', sans-serif;\\n            background-color: #f9f9f9;\\n            color: #333;\\n            padding: 2em;\\n            line-height: 1.6;\\n        }\\n        h1 {\\n            color: #004080;\\n            text-align: center;\\n        }\\n        .namespace {\\n            margin-bottom: 2em;\\n            border: 1px solid #ccc;\\n            padding: 1em;\\n            background-color: #fff;\\n            box-shadow: 0 2px 5px rgba(0,0,0,0.1);\\n            border-radius: 8px;\\n        }\\n        .pod-name {\\n            font-weight: bold;\\n            margin-top: 1em;\\n            font-size: 1.2em;\\n            color: #2a5885;\\n        }\\n        .section-title {\\n            margin-top: 1em;\\n            color: #555;\\n            font-size: 1em;\\n            border-bottom: 1px solid #ddd;\\n        }\\n        pre {\\n            background-color: #272822;\\n            color: #f8f8f2;\\n            padding: 1em;\\n            overflow-x: auto;\\n            white-space: pre-wrap;\\n            border-radius: 4px;\\n        }\\n        .events {\\n            background-color: #fffaf0;\\n            border-left: 5px solid #ffa500;\\n            padding: 1em;\\n            margin-top: 1em;\\n        }\\n    \\u003c/style\\u003e\\n\\u003c/head\\u003e\\n\\u003cbody\\u003e\\n\\u003ch1\\u003eKubeflow Pod Diagnostics Report\\u003c/h1\\u003e\\n\\\"\\\"\\\")\\n\\n        for entry in report_data:\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003ePod: {entry[\\\"pod\\\"]} (Namespace: {entry[\\\"namespace\\\"]})\\u003c/div\\u003e\\\\n')\\n            if entry.get(\\\"describe\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eDescribe\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"describe\\\"]}\\u003c/pre\\u003e\\\\n')\\n            if entry.get(\\\"logs\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eLogs\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"logs\\\"]}\\u003c/pre\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003eEvents in Namespace: {ns}\\u003c/div\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"events\\\"\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/div\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"065e0dae-405c-48d9-be83-a747d80b8ae0\",\"--execution_id\",\"141\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/065e0dae-405c-48d9-be83-a747d80b8ae0/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n    report_data = []\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # 모든 Pod 가져오기\\n    pods_output = subprocess.getoutput(\\n        \\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\"\\n    )\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    for line in pod_lines:\\n        if not line.strip():\\n            continue\\n        try:\\n            namespace, pod_name = line.split()\\n        except ValueError:\\n            continue\\n\\n        describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n        logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n        matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n        matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n        if matched_describe or matched_logs:\\n            report_data.append({\\n                \\\"namespace\\\": namespace,\\n                \\\"pod\\\": pod_name,\\n                \\\"describe\\\": matched_describe,\\n                \\\"logs\\\": matched_logs\\n            })\\n\\n        if namespace not in seen_namespaces:\\n            events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n            matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n            if matched_events:\\n                namespace_events[namespace] = matched_events\\n            seen_namespaces.add(namespace)\\n\\n    # HTML 작성\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(\\\"\\\"\\\"\\n\\u003c!DOCTYPE html\\u003e\\n\\u003chtml lang=\\\"en\\\"\\u003e\\n\\u003chead\\u003e\\n    \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n    \\u003ctitle\\u003eKubeflow Log Report\\u003c/title\\u003e\\n    \\u003cstyle\\u003e\\n        body {\\n            font-family: 'Segoe UI', sans-serif;\\n            background-color: #f9f9f9;\\n            color: #333;\\n            padding: 2em;\\n            line-height: 1.6;\\n        }\\n        h1 {\\n            color: #004080;\\n            text-align: center;\\n        }\\n        .namespace {\\n            margin-bottom: 2em;\\n            border: 1px solid #ccc;\\n            padding: 1em;\\n            background-color: #fff;\\n            box-shadow: 0 2px 5px rgba(0,0,0,0.1);\\n            border-radius: 8px;\\n        }\\n        .pod-name {\\n            font-weight: bold;\\n            margin-top: 1em;\\n            font-size: 1.2em;\\n            color: #2a5885;\\n        }\\n        .section-title {\\n            margin-top: 1em;\\n            color: #555;\\n            font-size: 1em;\\n            border-bottom: 1px solid #ddd;\\n        }\\n        pre {\\n            background-color: #272822;\\n            color: #f8f8f2;\\n            padding: 1em;\\n            overflow-x: auto;\\n            white-space: pre-wrap;\\n            border-radius: 4px;\\n        }\\n        .events {\\n            background-color: #fffaf0;\\n            border-left: 5px solid #ffa500;\\n            padding: 1em;\\n            margin-top: 1em;\\n        }\\n    \\u003c/style\\u003e\\n\\u003c/head\\u003e\\n\\u003cbody\\u003e\\n\\u003ch1\\u003eKubeflow Pod Diagnostics Report\\u003c/h1\\u003e\\n\\\"\\\"\\\")\\n\\n        for entry in report_data:\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003ePod: {entry[\\\"pod\\\"]} (Namespace: {entry[\\\"namespace\\\"]})\\u003c/div\\u003e\\\\n')\\n            if entry.get(\\\"describe\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eDescribe\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"describe\\\"]}\\u003c/pre\\u003e\\\\n')\\n            if entry.get(\\\"logs\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eLogs\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"logs\\\"]}\\u003c/pre\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003eEvents in Namespace: {ns}\\u003c/div\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"events\\\"\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/div\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-nm76f/2025/05/08/pod-log-to-github-nm76f-system-container-impl-518703743"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"065e0dae-405c-48d9-be83-a747d80b8ae0\",\"--execution_id\",\"141\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/065e0dae-405c-48d9-be83-a747d80b8ae0/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n    report_data = []\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # 모든 Pod 가져오기\\n    pods_output = subprocess.getoutput(\\n        \\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\"\\n    )\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    for line in pod_lines:\\n        if not line.strip():\\n            continue\\n        try:\\n            namespace, pod_name = line.split()\\n        except ValueError:\\n            continue\\n\\n        describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n        logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n        matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n        matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n        if matched_describe or matched_logs:\\n            report_data.append({\\n                \\\"namespace\\\": namespace,\\n                \\\"pod\\\": pod_name,\\n                \\\"describe\\\": matched_describe,\\n                \\\"logs\\\": matched_logs\\n            })\\n\\n        if namespace not in seen_namespaces:\\n            events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n            matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n            if matched_events:\\n                namespace_events[namespace] = matched_events\\n            seen_namespaces.add(namespace)\\n\\n    # HTML 작성\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(\\\"\\\"\\\"\\n\\u003c!DOCTYPE html\\u003e\\n\\u003chtml lang=\\\"en\\\"\\u003e\\n\\u003chead\\u003e\\n    \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n    \\u003ctitle\\u003eKubeflow Log Report\\u003c/title\\u003e\\n    \\u003cstyle\\u003e\\n        body {\\n            font-family: 'Segoe UI', sans-serif;\\n            background-color: #f9f9f9;\\n            color: #333;\\n            padding: 2em;\\n            line-height: 1.6;\\n        }\\n        h1 {\\n            color: #004080;\\n            text-align: center;\\n        }\\n        .namespace {\\n            margin-bottom: 2em;\\n            border: 1px solid #ccc;\\n            padding: 1em;\\n            background-color: #fff;\\n            box-shadow: 0 2px 5px rgba(0,0,0,0.1);\\n            border-radius: 8px;\\n        }\\n        .pod-name {\\n            font-weight: bold;\\n            margin-top: 1em;\\n            font-size: 1.2em;\\n            color: #2a5885;\\n        }\\n        .section-title {\\n            margin-top: 1em;\\n            color: #555;\\n            font-size: 1em;\\n            border-bottom: 1px solid #ddd;\\n        }\\n        pre {\\n            background-color: #272822;\\n            color: #f8f8f2;\\n            padding: 1em;\\n            overflow-x: auto;\\n            white-space: pre-wrap;\\n            border-radius: 4px;\\n        }\\n        .events {\\n            background-color: #fffaf0;\\n            border-left: 5px solid #ffa500;\\n            padding: 1em;\\n            margin-top: 1em;\\n        }\\n    \\u003c/style\\u003e\\n\\u003c/head\\u003e\\n\\u003cbody\\u003e\\n\\u003ch1\\u003eKubeflow Pod Diagnostics Report\\u003c/h1\\u003e\\n\\\"\\\"\\\")\\n\\n        for entry in report_data:\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003ePod: {entry[\\\"pod\\\"]} (Namespace: {entry[\\\"namespace\\\"]})\\u003c/div\\u003e\\\\n')\\n            if entry.get(\\\"describe\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eDescribe\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"describe\\\"]}\\u003c/pre\\u003e\\\\n')\\n            if entry.get(\\\"logs\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eLogs\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"logs\\\"]}\\u003c/pre\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003eEvents in Namespace: {ns}\\u003c/div\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"events\\\"\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/div\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"065e0dae-405c-48d9-be83-a747d80b8ae0\",\"--execution_id\",\"141\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/065e0dae-405c-48d9-be83-a747d80b8ae0/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n    report_data = []\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # 모든 Pod 가져오기\\n    pods_output = subprocess.getoutput(\\n        \\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\"\\n    )\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    for line in pod_lines:\\n        if not line.strip():\\n            continue\\n        try:\\n            namespace, pod_name = line.split()\\n        except ValueError:\\n            continue\\n\\n        describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n        logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n        matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n        matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n        if matched_describe or matched_logs:\\n            report_data.append({\\n                \\\"namespace\\\": namespace,\\n                \\\"pod\\\": pod_name,\\n                \\\"describe\\\": matched_describe,\\n                \\\"logs\\\": matched_logs\\n            })\\n\\n        if namespace not in seen_namespaces:\\n            events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n            matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n            if matched_events:\\n                namespace_events[namespace] = matched_events\\n            seen_namespaces.add(namespace)\\n\\n    # HTML 작성\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(\\\"\\\"\\\"\\n\\u003c!DOCTYPE html\\u003e\\n\\u003chtml lang=\\\"en\\\"\\u003e\\n\\u003chead\\u003e\\n    \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n    \\u003ctitle\\u003eKubeflow Log Report\\u003c/title\\u003e\\n    \\u003cstyle\\u003e\\n        body {\\n            font-family: 'Segoe UI', sans-serif;\\n            background-color: #f9f9f9;\\n            color: #333;\\n            padding: 2em;\\n            line-height: 1.6;\\n        }\\n        h1 {\\n            color: #004080;\\n            text-align: center;\\n        }\\n        .namespace {\\n            margin-bottom: 2em;\\n            border: 1px solid #ccc;\\n            padding: 1em;\\n            background-color: #fff;\\n            box-shadow: 0 2px 5px rgba(0,0,0,0.1);\\n            border-radius: 8px;\\n        }\\n        .pod-name {\\n            font-weight: bold;\\n            margin-top: 1em;\\n            font-size: 1.2em;\\n            color: #2a5885;\\n        }\\n        .section-title {\\n            margin-top: 1em;\\n            color: #555;\\n            font-size: 1em;\\n            border-bottom: 1px solid #ddd;\\n        }\\n        pre {\\n            background-color: #272822;\\n            color: #f8f8f2;\\n            padding: 1em;\\n            overflow-x: auto;\\n            white-space: pre-wrap;\\n            border-radius: 4px;\\n        }\\n        .events {\\n            background-color: #fffaf0;\\n            border-left: 5px solid #ffa500;\\n            padding: 1em;\\n            margin-top: 1em;\\n        }\\n    \\u003c/style\\u003e\\n\\u003c/head\\u003e\\n\\u003cbody\\u003e\\n\\u003ch1\\u003eKubeflow Pod Diagnostics Report\\u003c/h1\\u003e\\n\\\"\\\"\\\")\\n\\n        for entry in report_data:\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003ePod: {entry[\\\"pod\\\"]} (Namespace: {entry[\\\"namespace\\\"]})\\u003c/div\\u003e\\\\n')\\n            if entry.get(\\\"describe\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eDescribe\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"describe\\\"]}\\u003c/pre\\u003e\\\\n')\\n            if entry.get(\\\"logs\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eLogs\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"logs\\\"]}\\u003c/pre\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003eEvents in Namespace: {ns}\\u003c/div\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"events\\\"\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/div\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-nm76f/2025/05/08/pod-log-to-github-nm76f-system-container-impl-518703743"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"065e0dae-405c-48d9-be83-a747d80b8ae0\",\"--execution_id\",\"141\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/065e0dae-405c-48d9-be83-a747d80b8ae0/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n    report_data = []\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # 모든 Pod 가져오기\\n    pods_output = subprocess.getoutput(\\n        \\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\"\\n    )\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    for line in pod_lines:\\n        if not line.strip():\\n            continue\\n        try:\\n            namespace, pod_name = line.split()\\n        except ValueError:\\n            continue\\n\\n        describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n        logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n        matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n        matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n        if matched_describe or matched_logs:\\n            report_data.append({\\n                \\\"namespace\\\": namespace,\\n                \\\"pod\\\": pod_name,\\n                \\\"describe\\\": matched_describe,\\n                \\\"logs\\\": matched_logs\\n            })\\n\\n        if namespace not in seen_namespaces:\\n            events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n            matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n            if matched_events:\\n                namespace_events[namespace] = matched_events\\n            seen_namespaces.add(namespace)\\n\\n    # HTML 작성\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(\\\"\\\"\\\"\\n\\u003c!DOCTYPE html\\u003e\\n\\u003chtml lang=\\\"en\\\"\\u003e\\n\\u003chead\\u003e\\n    \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n    \\u003ctitle\\u003eKubeflow Log Report\\u003c/title\\u003e\\n    \\u003cstyle\\u003e\\n        body {\\n            font-family: 'Segoe UI', sans-serif;\\n            background-color: #f9f9f9;\\n            color: #333;\\n            padding: 2em;\\n            line-height: 1.6;\\n        }\\n        h1 {\\n            color: #004080;\\n            text-align: center;\\n        }\\n        .namespace {\\n            margin-bottom: 2em;\\n            border: 1px solid #ccc;\\n            padding: 1em;\\n            background-color: #fff;\\n            box-shadow: 0 2px 5px rgba(0,0,0,0.1);\\n            border-radius: 8px;\\n        }\\n        .pod-name {\\n            font-weight: bold;\\n            margin-top: 1em;\\n            font-size: 1.2em;\\n            color: #2a5885;\\n        }\\n        .section-title {\\n            margin-top: 1em;\\n            color: #555;\\n            font-size: 1em;\\n            border-bottom: 1px solid #ddd;\\n        }\\n        pre {\\n            background-color: #272822;\\n            color: #f8f8f2;\\n            padding: 1em;\\n            overflow-x: auto;\\n            white-space: pre-wrap;\\n            border-radius: 4px;\\n        }\\n        .events {\\n            background-color: #fffaf0;\\n            border-left: 5px solid #ffa500;\\n            padding: 1em;\\n            margin-top: 1em;\\n        }\\n    \\u003c/style\\u003e\\n\\u003c/head\\u003e\\n\\u003cbody\\u003e\\n\\u003ch1\\u003eKubeflow Pod Diagnostics Report\\u003c/h1\\u003e\\n\\\"\\\"\\\")\\n\\n        for entry in report_data:\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003ePod: {entry[\\\"pod\\\"]} (Namespace: {entry[\\\"namespace\\\"]})\\u003c/div\\u003e\\\\n')\\n            if entry.get(\\\"describe\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eDescribe\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"describe\\\"]}\\u003c/pre\\u003e\\\\n')\\n            if entry.get(\\\"logs\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eLogs\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"logs\\\"]}\\u003c/pre\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003eEvents in Namespace: {ns}\\u003c/div\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"events\\\"\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/div\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
              except ValueError:
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"065e0dae-405c-48d9-be83-a747d80b8ae0\",\"--execution_id\",\"141\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/065e0dae-405c-48d9-be83-a747d80b8ae0/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n    report_data = []\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # 모든 Pod 가져오기\\n    pods_output = subprocess.getoutput(\\n        \\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\"\\n    )\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    for line in pod_lines:\\n        if not line.strip():\\n            continue\\n        try:\\n            namespace, pod_name = line.split()\\n        except ValueError:\\n            continue\\n\\n        describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n        logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n        matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n        matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n        if matched_describe or matched_logs:\\n            report_data.append({\\n                \\\"namespace\\\": namespace,\\n                \\\"pod\\\": pod_name,\\n                \\\"describe\\\": matched_describe,\\n                \\\"logs\\\": matched_logs\\n            })\\n\\n        if namespace not in seen_namespaces:\\n            events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n            matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n            if matched_events:\\n                namespace_events[namespace] = matched_events\\n            seen_namespaces.add(namespace)\\n\\n    # HTML 작성\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(\\\"\\\"\\\"\\n\\u003c!DOCTYPE html\\u003e\\n\\u003chtml lang=\\\"en\\\"\\u003e\\n\\u003chead\\u003e\\n    \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n    \\u003ctitle\\u003eKubeflow Log Report\\u003c/title\\u003e\\n    \\u003cstyle\\u003e\\n        body {\\n            font-family: 'Segoe UI', sans-serif;\\n            background-color: #f9f9f9;\\n            color: #333;\\n            padding: 2em;\\n            line-height: 1.6;\\n        }\\n        h1 {\\n            color: #004080;\\n            text-align: center;\\n        }\\n        .namespace {\\n            margin-bottom: 2em;\\n            border: 1px solid #ccc;\\n            padding: 1em;\\n            background-color: #fff;\\n            box-shadow: 0 2px 5px rgba(0,0,0,0.1);\\n            border-radius: 8px;\\n        }\\n        .pod-name {\\n            font-weight: bold;\\n            margin-top: 1em;\\n            font-size: 1.2em;\\n            color: #2a5885;\\n        }\\n        .section-title {\\n            margin-top: 1em;\\n            color: #555;\\n            font-size: 1em;\\n            border-bottom: 1px solid #ddd;\\n        }\\n        pre {\\n            background-color: #272822;\\n            color: #f8f8f2;\\n            padding: 1em;\\n            overflow-x: auto;\\n            white-space: pre-wrap;\\n            border-radius: 4px;\\n        }\\n        .events {\\n            background-color: #fffaf0;\\n            border-left: 5px solid #ffa500;\\n            padding: 1em;\\n            margin-top: 1em;\\n        }\\n    \\u003c/style\\u003e\\n\\u003c/head\\u003e\\n\\u003cbody\\u003e\\n\\u003ch1\\u003eKubeflow Pod Diagnostics Report\\u003c/h1\\u003e\\n\\\"\\\"\\\")\\n\\n        for entry in report_data:\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003ePod: {entry[\\\"pod\\\"]} (Namespace: {entry[\\\"namespace\\\"]})\\u003c/div\\u003e\\\\n')\\n            if entry.get(\\\"describe\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eDescribe\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"describe\\\"]}\\u003c/pre\\u003e\\\\n')\\n            if entry.get(\\\"logs\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eLogs\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"logs\\\"]}\\u003c/pre\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003eEvents in Namespace: {ns}\\u003c/div\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"events\\\"\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/div\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-nm76f/2025/05/08/pod-log-to-github-nm76f-system-container-impl-518703743"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"065e0dae-405c-48d9-be83-a747d80b8ae0\",\"--execution_id\",\"141\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/065e0dae-405c-48d9-be83-a747d80b8ae0/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n    report_data = []\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # 모든 Pod 가져오기\\n    pods_output = subprocess.getoutput(\\n        \\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\"\\n    )\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    for line in pod_lines:\\n        if not line.strip():\\n            continue\\n        try:\\n            namespace, pod_name = line.split()\\n        except ValueError:\\n            continue\\n\\n        describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n        logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n        matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n        matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n        if matched_describe or matched_logs:\\n            report_data.append({\\n                \\\"namespace\\\": namespace,\\n                \\\"pod\\\": pod_name,\\n                \\\"describe\\\": matched_describe,\\n                \\\"logs\\\": matched_logs\\n            })\\n\\n        if namespace not in seen_namespaces:\\n            events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n            matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n            if matched_events:\\n                namespace_events[namespace] = matched_events\\n            seen_namespaces.add(namespace)\\n\\n    # HTML 작성\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(\\\"\\\"\\\"\\n\\u003c!DOCTYPE html\\u003e\\n\\u003chtml lang=\\\"en\\\"\\u003e\\n\\u003chead\\u003e\\n    \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n    \\u003ctitle\\u003eKubeflow Log Report\\u003c/title\\u003e\\n    \\u003cstyle\\u003e\\n        body {\\n            font-family: 'Segoe UI', sans-serif;\\n            background-color: #f9f9f9;\\n            color: #333;\\n            padding: 2em;\\n            line-height: 1.6;\\n        }\\n        h1 {\\n            color: #004080;\\n            text-align: center;\\n        }\\n        .namespace {\\n            margin-bottom: 2em;\\n            border: 1px solid #ccc;\\n            padding: 1em;\\n            background-color: #fff;\\n            box-shadow: 0 2px 5px rgba(0,0,0,0.1);\\n            border-radius: 8px;\\n        }\\n        .pod-name {\\n            font-weight: bold;\\n            margin-top: 1em;\\n            font-size: 1.2em;\\n            color: #2a5885;\\n        }\\n        .section-title {\\n            margin-top: 1em;\\n            color: #555;\\n            font-size: 1em;\\n            border-bottom: 1px solid #ddd;\\n        }\\n        pre {\\n            background-color: #272822;\\n            color: #f8f8f2;\\n            padding: 1em;\\n            overflow-x: auto;\\n            white-space: pre-wrap;\\n            border-radius: 4px;\\n        }\\n        .events {\\n            background-color: #fffaf0;\\n            border-left: 5px solid #ffa500;\\n            padding: 1em;\\n            margin-top: 1em;\\n        }\\n    \\u003c/style\\u003e\\n\\u003c/head\\u003e\\n\\u003cbody\\u003e\\n\\u003ch1\\u003eKubeflow Pod Diagnostics Report\\u003c/h1\\u003e\\n\\\"\\\"\\\")\\n\\n        for entry in report_data:\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003ePod: {entry[\\\"pod\\\"]} (Namespace: {entry[\\\"namespace\\\"]})\\u003c/div\\u003e\\\\n')\\n            if entry.get(\\\"describe\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eDescribe\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"describe\\\"]}\\u003c/pre\\u003e\\\\n')\\n            if entry.get(\\\"logs\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eLogs\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"logs\\\"]}\\u003c/pre\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003eEvents in Namespace: {ns}\\u003c/div\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"events\\\"\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/div\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}

Logs (filtered)

time="2025-05-08T06:14:20.548Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T06:14:21.488Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-nm76f-system-dag-driver-1198680945 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T06:12:49.218Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-nqq5c-system-container-driver-1096903435 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"137"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","6949ddbb-a13c-43eb-aa2b-187a4c1e4f15","--dag_execution_id","137","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-nqq5c/2025/05/08/pod-log-to-github-nqq5c-system-container-driver-1096903435"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"137"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","6949ddbb-a13c-43eb-aa2b-187a4c1e4f15","--dag_execution_id","137","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-nqq5c/2025/05/08/pod-log-to-github-nqq5c-system-container-driver-1096903435"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","collect_filtered_logs"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \"crash\"):\n    import shutil\n    import json\n    import subprocess\n    import os\n\n    tmp_report_path = \"/tmp/report.html\"\n    seen_namespaces = set()\n    namespace_events = {}\n\n    # kubectl 설치 방법 변경\n    subprocess.run([\"apt-get\", \"update\", \"-y\"], check=True)\n    subprocess.run([ \n        \"curl\", \"-LO\", \"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\"\n    ], check=True)\n    subprocess.run([\"chmod\", \"+x\", \"kubectl\"], check=True)\n    subprocess.run([\"mv\", \"kubectl\", \"/usr/local/bin/\"], check=True)\n\n    # Get all pods in all namespaces\n    pods_output = subprocess.getoutput(\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\")\n    pod_lines = pods_output.strip().split(\"\\n\")\n\n    with open(tmp_report_path, \"w\") as f:\n        f.write(f\"\u003chtml\u003e\u003cbody\u003e\u003ch1\u003e장애 복기 리포트 (keyword: `{keyword}`)\u003c/h1\u003e\\n\")\n\n        for line in pod_lines:\n            if not line.strip():\n                continue\n            try:\n                namespace, pod_name = line.split()\n            except ValueError:\n                continue\n\n            describe = subprocess.getoutput(f\"kubectl describe pod {pod_name} -n {namespace}\")\n            logs = subprocess.getoutput(f\"kubectl logs {pod_name} -n {namespace}\")\n\n            matched_describe = \"\\n\".join([l for l in describe.splitlines() if keyword in l])\n            matched_logs = \"\\n\".join([l for l in logs.splitlines() if keyword in l])\n\n            if matched_describe or matched_logs:\n                f.write(f\"\u003ch2\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\u003c/h2\u003e\\n\")\n                if matched_describe:\n                    f.write(f\"\u003ch3\u003eDescribe\u003c/h3\u003e\u003cpre\u003e{matched_describe}\u003c/pre\u003e\\n\")\n                if matched_logs:\n                    f.write(f\"\u003ch3\u003eLogs\u003c/h3\u003e\u003cpre\u003e{matched_logs}\u003c/pre\u003e\\n\")\n\n            if namespace not in seen_namespaces:\n                events = subprocess.getoutput(f\"kubectl get events -n {namespace}\")\n                matched_events = \"\\n\".join([l for l in events.splitlines() if keyword in l])\n                if matched_events:\n                    namespace_events[namespace] = matched_events\n                seen_namespaces.add(namespace)\n\n        # Write filtered events grouped by namespace\n        for ns, events in namespace_events.items():\n            f.write(f\"\u003ch3\u003eEvents in Namespace: `{ns}`\u003c/h3\u003e\u003cpre\u003e{events}\u003c/pre\u003e\\n\")\n\n        f.write(\"\u003c/body\u003e\u003c/html\u003e\")\n\n    subprocess.run([\"cp\", tmp_report_path, output_report.path], check=True)\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"137"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","6949ddbb-a13c-43eb-aa2b-187a4c1e4f15","--dag_execution_id","137","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-nqq5c/2025/05/08/pod-log-to-github-nqq5c-system-container-driver-1096903435"}}}

Logs (filtered)

time="2025-05-08T06:04:50.728Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-nqq5c-system-container-driver-4290442412 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"137"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","6949ddbb-a13c-43eb-aa2b-187a4c1e4f15","--dag_execution_id","137","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-nqq5c/2025/05/08/pod-log-to-github-nqq5c-system-container-driver-4290442412"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"137"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","6949ddbb-a13c-43eb-aa2b-187a4c1e4f15","--dag_execution_id","137","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-nqq5c/2025/05/08/pod-log-to-github-nqq5c-system-container-driver-4290442412"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str\n):\n\n    import subprocess\n    import shutil\n    import os\n    from datetime import datetime\n\n    os.environ[\"GIT_ASKPASS\"] = \"echo\"\n\n    report_path = report_file.path\n    print(\"Checking if report exists at:\", report_path)\n    if not os.path.exists(report_path):\n        raise FileNotFoundError(f\"{report_path} does not exist\")\n\n    # HTML 파일 이름 만들기\n    timestamp = datetime.now().strftime(\"report-%Y%m%d-%H%M%S.html\")\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n    report_target_path = os.path.join(repo_dir, timestamp)\n\n    # Git clone\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    shutil.copyfile(report_path, report_target_path)\n\n    # Git config\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    # Git commit \u0026 push\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", f\"Add {timestamp}\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(f\"Report committed and pushed to GitHub as {timestamp}.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"137"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","6949ddbb-a13c-43eb-aa2b-187a4c1e4f15","--dag_execution_id","137","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-nqq5c/2025/05/08/pod-log-to-github-nqq5c-system-container-driver-4290442412"}}}

Logs (filtered)

time="2025-05-08T06:06:24.921Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-nqq5c-system-container-impl-2211538493 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"6949ddbb-a13c-43eb-aa2b-187a4c1e4f15\",\"--execution_id\",\"138\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/6949ddbb-a13c-43eb-aa2b-187a4c1e4f15/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-nqq5c/2025/05/08/pod-log-to-github-nqq5c-system-container-impl-2211538493"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"6949ddbb-a13c-43eb-aa2b-187a4c1e4f15\",\"--execution_id\",\"138\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/6949ddbb-a13c-43eb-aa2b-187a4c1e4f15/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"6949ddbb-a13c-43eb-aa2b-187a4c1e4f15\",\"--execution_id\",\"138\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/6949ddbb-a13c-43eb-aa2b-187a4c1e4f15/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-nqq5c/2025/05/08/pod-log-to-github-nqq5c-system-container-impl-2211538493"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"6949ddbb-a13c-43eb-aa2b-187a4c1e4f15\",\"--execution_id\",\"138\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/6949ddbb-a13c-43eb-aa2b-187a4c1e4f15/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"6949ddbb-a13c-43eb-aa2b-187a4c1e4f15\",\"--execution_id\",\"138\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/6949ddbb-a13c-43eb-aa2b-187a4c1e4f15/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-nqq5c/2025/05/08/pod-log-to-github-nqq5c-system-container-impl-2211538493"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"6949ddbb-a13c-43eb-aa2b-187a4c1e4f15\",\"--execution_id\",\"138\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/6949ddbb-a13c-43eb-aa2b-187a4c1e4f15/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
                  except ValueError:
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"6949ddbb-a13c-43eb-aa2b-187a4c1e4f15\",\"--execution_id\",\"138\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/6949ddbb-a13c-43eb-aa2b-187a4c1e4f15/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-nqq5c/2025/05/08/pod-log-to-github-nqq5c-system-container-impl-2211538493"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"6949ddbb-a13c-43eb-aa2b-187a4c1e4f15\",\"--execution_id\",\"138\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/6949ddbb-a13c-43eb-aa2b-187a4c1e4f15/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}

Logs (filtered)

time="2025-05-08T06:06:11.062Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T06:06:11.993Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-nqq5c-system-container-impl-3380551554 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"6949ddbb-a13c-43eb-aa2b-187a4c1e4f15\",\"--execution_id\",\"139\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"23\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/6949ddbb-a13c-43eb-aa2b-187a4c1e4f15/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-nqq5c/2025/05/08/pod-log-to-github-nqq5c-system-container-impl-3380551554"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"6949ddbb-a13c-43eb-aa2b-187a4c1e4f15\",\"--execution_id\",\"139\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"23\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/6949ddbb-a13c-43eb-aa2b-187a4c1e4f15/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"6949ddbb-a13c-43eb-aa2b-187a4c1e4f15\",\"--execution_id\",\"139\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"23\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/6949ddbb-a13c-43eb-aa2b-187a4c1e4f15/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-nqq5c/2025/05/08/pod-log-to-github-nqq5c-system-container-impl-3380551554"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"6949ddbb-a13c-43eb-aa2b-187a4c1e4f15\",\"--execution_id\",\"139\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"23\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/6949ddbb-a13c-43eb-aa2b-187a4c1e4f15/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"6949ddbb-a13c-43eb-aa2b-187a4c1e4f15\",\"--execution_id\",\"139\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"23\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/6949ddbb-a13c-43eb-aa2b-187a4c1e4f15/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-nqq5c/2025/05/08/pod-log-to-github-nqq5c-system-container-impl-3380551554"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"6949ddbb-a13c-43eb-aa2b-187a4c1e4f15\",\"--execution_id\",\"139\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"23\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/6949ddbb-a13c-43eb-aa2b-187a4c1e4f15/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
              raise FileNotFoundError(f"{report_path} does not exist")
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"6949ddbb-a13c-43eb-aa2b-187a4c1e4f15\",\"--execution_id\",\"139\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"23\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/6949ddbb-a13c-43eb-aa2b-187a4c1e4f15/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-nqq5c/2025/05/08/pod-log-to-github-nqq5c-system-container-impl-3380551554"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"6949ddbb-a13c-43eb-aa2b-187a4c1e4f15\",\"--execution_id\",\"139\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"23\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/6949ddbb-a13c-43eb-aa2b-187a4c1e4f15/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}

Logs (filtered)

time="2025-05-08T06:06:45.061Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T06:06:45.999Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-nqq5c-system-dag-driver-2712913191 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T06:04:40.625Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-nqs86-system-container-driver-331213999 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T00:44:36.823Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-nqs86-system-container-driver-812905276 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"90"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","04f074cb-5cba-4c19-aa05-b691774e5065","--dag_execution_id","90","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-nqs86/2025/05/08/pod-log-to-github-nqs86-system-container-driver-812905276"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"90"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","04f074cb-5cba-4c19-aa05-b691774e5065","--dag_execution_id","90","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-nqs86/2025/05/08/pod-log-to-github-nqs86-system-container-driver-812905276"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str = \"main\"\n):\n    import os\n    import subprocess\n\n\n    report_path = report_file.path\n    print(\"Checking if report exists at:\", report_path)\n    if not os.path.exists(report_path):\n        raise FileNotFoundError(f\"{report_path} does not exist\")\n\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    subprocess.run([\"cp\", report_path, os.path.join(repo_dir, \"report.md\")], check=True)\n\n    # Git user config\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", \"Add report\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(\"Report committed and pushed to GitHub.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"90"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","04f074cb-5cba-4c19-aa05-b691774e5065","--dag_execution_id","90","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-nqs86/2025/05/08/pod-log-to-github-nqs86-system-container-driver-812905276"}}}

Logs (filtered)

time="2025-05-08T00:45:54.629Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-nqs86-system-container-impl-2763467937 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T00:45:40.978Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T00:45:41.918Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-nqs86-system-dag-driver-3268095479 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T00:44:26.632Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-phkw4-system-container-driver-4158966025 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"110"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","29408b4a-7ecd-40ad-bc54-fb0425545642","--dag_execution_id","110","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-phkw4/2025/05/08/pod-log-to-github-phkw4-system-container-driver-4158966025"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"110"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","29408b4a-7ecd-40ad-bc54-fb0425545642","--dag_execution_id","110","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-phkw4/2025/05/08/pod-log-to-github-phkw4-system-container-driver-4158966025"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","collect_filtered_logs"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \"crash\"):\n    import shutil\n    import json\n    import subprocess\n    import os\n\n    tmp_report_path = \"/tmp/report.html\"\n    seen_namespaces = set()\n    namespace_events = {}\n\n    # kubectl 설치 방법 변경\n    subprocess.run([\"apt-get\", \"update\", \"-y\"], check=True)\n    subprocess.run([ \n        \"curl\", \"-LO\", \"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\"\n    ], check=True)\n    subprocess.run([\"chmod\", \"+x\", \"kubectl\"], check=True)\n    subprocess.run([\"mv\", \"kubectl\", \"/usr/local/bin/\"], check=True)\n\n    # Get all pods in all namespaces\n    pods_output = subprocess.getoutput(\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\")\n    pod_lines = pods_output.strip().split(\"\\n\")\n\n    with open(tmp_report_path, \"w\") as f:\n        f.write(f\"\u003chtml\u003e\u003cbody\u003e\u003ch1\u003e장애 복기 리포트 (keyword: `{keyword}`)\u003c/h1\u003e\\n\")\n\n        for line in pod_lines:\n            if not line.strip():\n                continue\n            try:\n                namespace, pod_name = line.split()\n            except ValueError:\n                continue\n\n            describe = subprocess.getoutput(f\"kubectl describe pod {pod_name} -n {namespace}\")\n            logs = subprocess.getoutput(f\"kubectl logs {pod_name} -n {namespace}\")\n\n            matched_describe = \"\\n\".join([l for l in describe.splitlines() if keyword in l])\n            matched_logs = \"\\n\".join([l for l in logs.splitlines() if keyword in l])\n\n            if matched_describe or matched_logs:\n                f.write(f\"\u003ch2\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\u003c/h2\u003e\\n\")\n                if matched_describe:\n                    f.write(f\"\u003ch3\u003eDescribe\u003c/h3\u003e\u003cpre\u003e{matched_describe}\u003c/pre\u003e\\n\")\n                if matched_logs:\n                    f.write(f\"\u003ch3\u003eLogs\u003c/h3\u003e\u003cpre\u003e{matched_logs}\u003c/pre\u003e\\n\")\n\n            if namespace not in seen_namespaces:\n                events = subprocess.getoutput(f\"kubectl get events -n {namespace}\")\n                matched_events = \"\\n\".join([l for l in events.splitlines() if keyword in l])\n                if matched_events:\n                    namespace_events[namespace] = matched_events\n                seen_namespaces.add(namespace)\n\n        # Write filtered events grouped by namespace\n        for ns, events in namespace_events.items():\n            f.write(f\"\u003ch3\u003eEvents in Namespace: `{ns}`\u003c/h3\u003e\u003cpre\u003e{events}\u003c/pre\u003e\\n\")\n\n        f.write(\"\u003c/body\u003e\u003c/html\u003e\")\n\n    subprocess.run([\"cp\", tmp_report_path, output_report.path], check=True)\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"110"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","29408b4a-7ecd-40ad-bc54-fb0425545642","--dag_execution_id","110","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-phkw4/2025/05/08/pod-log-to-github-phkw4-system-container-driver-4158966025"}}}

Logs (filtered)

time="2025-05-08T04:51:55.124Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-phkw4-system-container-driver-89389142 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"110"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","29408b4a-7ecd-40ad-bc54-fb0425545642","--dag_execution_id","110","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-phkw4/2025/05/08/pod-log-to-github-phkw4-system-container-driver-89389142"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"110"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","29408b4a-7ecd-40ad-bc54-fb0425545642","--dag_execution_id","110","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-phkw4/2025/05/08/pod-log-to-github-phkw4-system-container-driver-89389142"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str = \"main\"\n):\n\n    import subprocess\n    import shutil\n    import os\n\n    report_path = report_file.path\n    print(\"Checking if report exists at:\", report_path)\n    if not os.path.exists(report_path):\n        raise FileNotFoundError(f\"{report_path} does not exist\")\n\n    # HTML 파일 이름 만들기\n    timestamp = datetime.now().strftime(\"report-%Y%m%d-%H%M%S.html\")\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n    report_target_path = os.path.join(repo_dir, timestamp)\n\n    # Git clone\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    shutil.copyfile(report_path, report_target_path)\n\n    # Git config\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    # Git commit \u0026 push\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", f\"Add {timestamp}\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(f\"Report committed and pushed to GitHub as {timestamp}.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"110"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","29408b4a-7ecd-40ad-bc54-fb0425545642","--dag_execution_id","110","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-phkw4/2025/05/08/pod-log-to-github-phkw4-system-container-driver-89389142"}}}

Logs (filtered)

time="2025-05-08T04:53:26.218Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-phkw4-system-container-impl-1729468608 (Namespace: kubeflow-user-example-com)

Describe (filtered)

Status:       Failed
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29408b4a-7ecd-40ad-bc54-fb0425545642\",\"--execution_id\",\"112\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"14\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29408b4a-7ecd-40ad-bc54-fb0425545642/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-phkw4/2025/05/08/pod-log-to-github-phkw4-system-container-impl-1729468608"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29408b4a-7ecd-40ad-bc54-fb0425545642\",\"--execution_id\",\"112\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"14\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29408b4a-7ecd-40ad-bc54-fb0425545642/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29408b4a-7ecd-40ad-bc54-fb0425545642\",\"--execution_id\",\"112\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"14\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29408b4a-7ecd-40ad-bc54-fb0425545642/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-phkw4/2025/05/08/pod-log-to-github-phkw4-system-container-impl-1729468608"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29408b4a-7ecd-40ad-bc54-fb0425545642\",\"--execution_id\",\"112\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"14\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29408b4a-7ecd-40ad-bc54-fb0425545642/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29408b4a-7ecd-40ad-bc54-fb0425545642\",\"--execution_id\",\"112\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"14\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29408b4a-7ecd-40ad-bc54-fb0425545642/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-phkw4/2025/05/08/pod-log-to-github-phkw4-system-container-impl-1729468608"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29408b4a-7ecd-40ad-bc54-fb0425545642\",\"--execution_id\",\"112\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"14\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29408b4a-7ecd-40ad-bc54-fb0425545642/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
              raise FileNotFoundError(f"{report_path} does not exist")
      Reason:       Error
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29408b4a-7ecd-40ad-bc54-fb0425545642\",\"--execution_id\",\"112\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"14\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29408b4a-7ecd-40ad-bc54-fb0425545642/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-phkw4/2025/05/08/pod-log-to-github-phkw4-system-container-impl-1729468608"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29408b4a-7ecd-40ad-bc54-fb0425545642\",\"--execution_id\",\"112\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"14\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29408b4a-7ecd-40ad-bc54-fb0425545642/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}

Logs (filtered)

Traceback (most recent call last):
F0508 04:53:43.239970      22 main.go:49] failed to execute component: exit status 1
time="2025-05-08T04:53:43.340Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T04:53:44.284Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-phkw4-system-container-impl-2177164743 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29408b4a-7ecd-40ad-bc54-fb0425545642\",\"--execution_id\",\"111\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29408b4a-7ecd-40ad-bc54-fb0425545642/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-phkw4/2025/05/08/pod-log-to-github-phkw4-system-container-impl-2177164743"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29408b4a-7ecd-40ad-bc54-fb0425545642\",\"--execution_id\",\"111\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29408b4a-7ecd-40ad-bc54-fb0425545642/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29408b4a-7ecd-40ad-bc54-fb0425545642\",\"--execution_id\",\"111\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29408b4a-7ecd-40ad-bc54-fb0425545642/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-phkw4/2025/05/08/pod-log-to-github-phkw4-system-container-impl-2177164743"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29408b4a-7ecd-40ad-bc54-fb0425545642\",\"--execution_id\",\"111\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29408b4a-7ecd-40ad-bc54-fb0425545642/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29408b4a-7ecd-40ad-bc54-fb0425545642\",\"--execution_id\",\"111\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29408b4a-7ecd-40ad-bc54-fb0425545642/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-phkw4/2025/05/08/pod-log-to-github-phkw4-system-container-impl-2177164743"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29408b4a-7ecd-40ad-bc54-fb0425545642\",\"--execution_id\",\"111\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29408b4a-7ecd-40ad-bc54-fb0425545642/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
                  except ValueError:
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29408b4a-7ecd-40ad-bc54-fb0425545642\",\"--execution_id\",\"111\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29408b4a-7ecd-40ad-bc54-fb0425545642/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-phkw4/2025/05/08/pod-log-to-github-phkw4-system-container-impl-2177164743"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"29408b4a-7ecd-40ad-bc54-fb0425545642\",\"--execution_id\",\"111\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/29408b4a-7ecd-40ad-bc54-fb0425545642/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}

Logs (filtered)

time="2025-05-08T04:53:12.312Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T04:53:13.236Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-phkw4-system-dag-driver-3288213737 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T04:51:44.924Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-pm5rh-system-container-driver-2653658484 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"131"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","184dbb68-5668-4afe-a4eb-c6030a99d23c","--dag_execution_id","131","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-pm5rh/2025/05/08/pod-log-to-github-pm5rh-system-container-driver-2653658484"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"131"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","184dbb68-5668-4afe-a4eb-c6030a99d23c","--dag_execution_id","131","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-pm5rh/2025/05/08/pod-log-to-github-pm5rh-system-container-driver-2653658484"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str = \"main\"\n):\n\n    import subprocess\n    import shutil\n    import os\n    from datetime import datetime\n\n    os.environ[\"GIT_ASKPASS\"] = \"echo\"\n\n    report_path = report_file.path\n    print(\"Checking if report exists at:\", report_path)\n    if not os.path.exists(report_path):\n        raise FileNotFoundError(f\"{report_path} does not exist\")\n\n    # HTML 파일 이름 만들기\n    timestamp = datetime.now().strftime(\"report-%Y%m%d-%H%M%S.html\")\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n    report_target_path = os.path.join(repo_dir, timestamp)\n\n    # Git clone\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    shutil.copyfile(report_path, report_target_path)\n\n    # Git config\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    # Git commit \u0026 push\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", f\"Add {timestamp}\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(f\"Report committed and pushed to GitHub as {timestamp}.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"131"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","184dbb68-5668-4afe-a4eb-c6030a99d23c","--dag_execution_id","131","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-pm5rh/2025/05/08/pod-log-to-github-pm5rh-system-container-driver-2653658484"}}}

Logs (filtered)

time="2025-05-08T05:56:16.810Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-pm5rh-system-container-driver-3249322595 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"131"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","184dbb68-5668-4afe-a4eb-c6030a99d23c","--dag_execution_id","131","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-pm5rh/2025/05/08/pod-log-to-github-pm5rh-system-container-driver-3249322595"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"131"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","184dbb68-5668-4afe-a4eb-c6030a99d23c","--dag_execution_id","131","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-pm5rh/2025/05/08/pod-log-to-github-pm5rh-system-container-driver-3249322595"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","collect_filtered_logs"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \"crash\"):\n    import shutil\n    import json\n    import subprocess\n    import os\n\n    tmp_report_path = \"/tmp/report.html\"\n    seen_namespaces = set()\n    namespace_events = {}\n\n    # kubectl 설치 방법 변경\n    subprocess.run([\"apt-get\", \"update\", \"-y\"], check=True)\n    subprocess.run([ \n        \"curl\", \"-LO\", \"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\"\n    ], check=True)\n    subprocess.run([\"chmod\", \"+x\", \"kubectl\"], check=True)\n    subprocess.run([\"mv\", \"kubectl\", \"/usr/local/bin/\"], check=True)\n\n    # Get all pods in all namespaces\n    pods_output = subprocess.getoutput(\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\")\n    pod_lines = pods_output.strip().split(\"\\n\")\n\n    with open(tmp_report_path, \"w\") as f:\n        f.write(f\"\u003chtml\u003e\u003cbody\u003e\u003ch1\u003e장애 복기 리포트 (keyword: `{keyword}`)\u003c/h1\u003e\\n\")\n\n        for line in pod_lines:\n            if not line.strip():\n                continue\n            try:\n                namespace, pod_name = line.split()\n            except ValueError:\n                continue\n\n            describe = subprocess.getoutput(f\"kubectl describe pod {pod_name} -n {namespace}\")\n            logs = subprocess.getoutput(f\"kubectl logs {pod_name} -n {namespace}\")\n\n            matched_describe = \"\\n\".join([l for l in describe.splitlines() if keyword in l])\n            matched_logs = \"\\n\".join([l for l in logs.splitlines() if keyword in l])\n\n            if matched_describe or matched_logs:\n                f.write(f\"\u003ch2\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\u003c/h2\u003e\\n\")\n                if matched_describe:\n                    f.write(f\"\u003ch3\u003eDescribe\u003c/h3\u003e\u003cpre\u003e{matched_describe}\u003c/pre\u003e\\n\")\n                if matched_logs:\n                    f.write(f\"\u003ch3\u003eLogs\u003c/h3\u003e\u003cpre\u003e{matched_logs}\u003c/pre\u003e\\n\")\n\n            if namespace not in seen_namespaces:\n                events = subprocess.getoutput(f\"kubectl get events -n {namespace}\")\n                matched_events = \"\\n\".join([l for l in events.splitlines() if keyword in l])\n                if matched_events:\n                    namespace_events[namespace] = matched_events\n                seen_namespaces.add(namespace)\n\n        # Write filtered events grouped by namespace\n        for ns, events in namespace_events.items():\n            f.write(f\"\u003ch3\u003eEvents in Namespace: `{ns}`\u003c/h3\u003e\u003cpre\u003e{events}\u003c/pre\u003e\\n\")\n\n        f.write(\"\u003c/body\u003e\u003c/html\u003e\")\n\n    subprocess.run([\"cp\", tmp_report_path, output_report.path], check=True)\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"131"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","184dbb68-5668-4afe-a4eb-c6030a99d23c","--dag_execution_id","131","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-pm5rh/2025/05/08/pod-log-to-github-pm5rh-system-container-driver-3249322595"}}}

Logs (filtered)

time="2025-05-08T05:54:42.642Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-pm5rh-system-container-impl-2488056613 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"184dbb68-5668-4afe-a4eb-c6030a99d23c\",\"--execution_id\",\"132\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/184dbb68-5668-4afe-a4eb-c6030a99d23c/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-pm5rh/2025/05/08/pod-log-to-github-pm5rh-system-container-impl-2488056613"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"184dbb68-5668-4afe-a4eb-c6030a99d23c\",\"--execution_id\",\"132\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/184dbb68-5668-4afe-a4eb-c6030a99d23c/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"184dbb68-5668-4afe-a4eb-c6030a99d23c\",\"--execution_id\",\"132\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/184dbb68-5668-4afe-a4eb-c6030a99d23c/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-pm5rh/2025/05/08/pod-log-to-github-pm5rh-system-container-impl-2488056613"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"184dbb68-5668-4afe-a4eb-c6030a99d23c\",\"--execution_id\",\"132\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/184dbb68-5668-4afe-a4eb-c6030a99d23c/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"184dbb68-5668-4afe-a4eb-c6030a99d23c\",\"--execution_id\",\"132\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/184dbb68-5668-4afe-a4eb-c6030a99d23c/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-pm5rh/2025/05/08/pod-log-to-github-pm5rh-system-container-impl-2488056613"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"184dbb68-5668-4afe-a4eb-c6030a99d23c\",\"--execution_id\",\"132\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/184dbb68-5668-4afe-a4eb-c6030a99d23c/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
                  except ValueError:
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"184dbb68-5668-4afe-a4eb-c6030a99d23c\",\"--execution_id\",\"132\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/184dbb68-5668-4afe-a4eb-c6030a99d23c/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-pm5rh/2025/05/08/pod-log-to-github-pm5rh-system-container-impl-2488056613"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"184dbb68-5668-4afe-a4eb-c6030a99d23c\",\"--execution_id\",\"132\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/184dbb68-5668-4afe-a4eb-c6030a99d23c/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}

Logs (filtered)

time="2025-05-08T05:56:03.011Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T05:56:03.949Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-pm5rh-system-container-impl-3177957402 (Namespace: kubeflow-user-example-com)

Describe (filtered)

Status:       Failed
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"184dbb68-5668-4afe-a4eb-c6030a99d23c\",\"--execution_id\",\"133\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"21\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/184dbb68-5668-4afe-a4eb-c6030a99d23c/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-pm5rh/2025/05/08/pod-log-to-github-pm5rh-system-container-impl-3177957402"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"184dbb68-5668-4afe-a4eb-c6030a99d23c\",\"--execution_id\",\"133\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"21\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/184dbb68-5668-4afe-a4eb-c6030a99d23c/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"184dbb68-5668-4afe-a4eb-c6030a99d23c\",\"--execution_id\",\"133\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"21\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/184dbb68-5668-4afe-a4eb-c6030a99d23c/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-pm5rh/2025/05/08/pod-log-to-github-pm5rh-system-container-impl-3177957402"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"184dbb68-5668-4afe-a4eb-c6030a99d23c\",\"--execution_id\",\"133\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"21\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/184dbb68-5668-4afe-a4eb-c6030a99d23c/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"184dbb68-5668-4afe-a4eb-c6030a99d23c\",\"--execution_id\",\"133\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"21\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/184dbb68-5668-4afe-a4eb-c6030a99d23c/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-pm5rh/2025/05/08/pod-log-to-github-pm5rh-system-container-impl-3177957402"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"184dbb68-5668-4afe-a4eb-c6030a99d23c\",\"--execution_id\",\"133\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"21\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/184dbb68-5668-4afe-a4eb-c6030a99d23c/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
              raise FileNotFoundError(f"{report_path} does not exist")
      Reason:       Error
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"184dbb68-5668-4afe-a4eb-c6030a99d23c\",\"--execution_id\",\"133\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"21\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/184dbb68-5668-4afe-a4eb-c6030a99d23c/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-pm5rh/2025/05/08/pod-log-to-github-pm5rh-system-container-impl-3177957402"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"184dbb68-5668-4afe-a4eb-c6030a99d23c\",\"--execution_id\",\"133\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"21\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/184dbb68-5668-4afe-a4eb-c6030a99d23c/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}

Logs (filtered)

remote: error: GH013: Repository rule violations found for refs/heads/report.        
error: failed to push some refs to 'https://github.com/Rosa1026/Kubeflow.git'
Traceback (most recent call last):
F0508 05:56:36.376650      24 main.go:49] failed to execute component: exit status 1
time="2025-05-08T05:56:37.039Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T05:56:37.984Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-pm5rh-system-dag-driver-647919199 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T05:54:32.535Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-qlsrg-system-container-driver-1950805462 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T07:40:52.743Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-qlsrg-system-container-impl-2629519936 (Namespace: kubeflow-user-example-com)

Describe (filtered)

Status:       Failed
      Reason:       Error

Logs (filtered)

Traceback (most recent call last):
F0507 07:41:08.297346      23 main.go:49] failed to execute component: exit status 1
time="2025-05-07T07:41:08.959Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-07T07:41:09.888Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-qlsrg-system-dag-driver-1478221602 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T07:40:42.725Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-r4rj2-system-container-driver-2593530976 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"67"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","76dbd840-2b5a-4399-9d75-456c61d05bec","--dag_execution_id","67","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-r4rj2/2025/05/07/pod-log-to-github-r4rj2-system-container-driver-2593530976"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"67"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","76dbd840-2b5a-4399-9d75-456c61d05bec","--dag_execution_id","67","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-r4rj2/2025/05/07/pod-log-to-github-r4rj2-system-container-driver-2593530976"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str = \"main\"\n):\n    import os\n    import subprocess\n\n\n    report_path = report_file.path\n    print(\"Checking if report exists at:\", report_path)\n    if not os.path.exists(report_path):\n        raise FileNotFoundError(f\"{report_path} does not exist\")\n\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    subprocess.run([\"cp\", report_path, os.path.join(repo_dir, \"report.md\")], check=True)\n\n    # Git user config\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", \"Add report\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(\"Report committed and pushed to GitHub.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"67"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","76dbd840-2b5a-4399-9d75-456c61d05bec","--dag_execution_id","67","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-r4rj2/2025/05/07/pod-log-to-github-r4rj2-system-container-driver-2593530976"}}}

Logs (filtered)

time="2025-05-07T08:19:09.023Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-07T08:19:09.023Z" level=error msg="cannot save parameter /tmp/outputs/pod-spec-patch" argo=true error="open /tmp/outputs/pod-spec-patch: no such file or directory"

Pod: pod-log-to-github-r4rj2-system-container-driver-2652044541 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T08:18:58.924Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-07T08:18:58.925Z" level=error msg="cannot save parameter /tmp/outputs/pod-spec-patch" argo=true error="open /tmp/outputs/pod-spec-patch: no such file or directory"

Pod: pod-log-to-github-r4rj2-system-dag-driver-3512469619 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T08:18:48.830Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-scgtw-system-container-driver-2581822818 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"128"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","01d67935-b0f8-4b00-b0de-e304235cd881","--dag_execution_id","128","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-scgtw/2025/05/08/pod-log-to-github-scgtw-system-container-driver-2581822818"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"128"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","01d67935-b0f8-4b00-b0de-e304235cd881","--dag_execution_id","128","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-scgtw/2025/05/08/pod-log-to-github-scgtw-system-container-driver-2581822818"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str = \"main\"\n):\n\n    import subprocess\n    import shutil\n    import os\n    from datetime import datetime\n\n    os.environ[\"GIT_ASKPASS\"] = \"echo\"\n\n    report_path = report_file.path\n    print(\"Checking if report exists at:\", report_path)\n    if not os.path.exists(report_path):\n        raise FileNotFoundError(f\"{report_path} does not exist\")\n\n    # HTML 파일 이름 만들기\n    timestamp = datetime.now().strftime(\"report-%Y%m%d-%H%M%S.html\")\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n    report_target_path = os.path.join(repo_dir, timestamp)\n\n    # Git clone\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    shutil.copyfile(report_path, report_target_path)\n\n    # Git config\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    # Git commit \u0026 push\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", f\"Add {timestamp}\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(f\"Report committed and pushed to GitHub as {timestamp}.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"128"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","01d67935-b0f8-4b00-b0de-e304235cd881","--dag_execution_id","128","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-scgtw/2025/05/08/pod-log-to-github-scgtw-system-container-driver-2581822818"}}}

Logs (filtered)

time="2025-05-08T05:52:39.425Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-scgtw-system-container-driver-420083653 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"128"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","01d67935-b0f8-4b00-b0de-e304235cd881","--dag_execution_id","128","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-scgtw/2025/05/08/pod-log-to-github-scgtw-system-container-driver-420083653"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"128"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","01d67935-b0f8-4b00-b0de-e304235cd881","--dag_execution_id","128","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-scgtw/2025/05/08/pod-log-to-github-scgtw-system-container-driver-420083653"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","collect_filtered_logs"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \"crash\"):\n    import shutil\n    import json\n    import subprocess\n    import os\n\n    tmp_report_path = \"/tmp/report.html\"\n    seen_namespaces = set()\n    namespace_events = {}\n\n    # kubectl 설치 방법 변경\n    subprocess.run([\"apt-get\", \"update\", \"-y\"], check=True)\n    subprocess.run([ \n        \"curl\", \"-LO\", \"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\"\n    ], check=True)\n    subprocess.run([\"chmod\", \"+x\", \"kubectl\"], check=True)\n    subprocess.run([\"mv\", \"kubectl\", \"/usr/local/bin/\"], check=True)\n\n    # Get all pods in all namespaces\n    pods_output = subprocess.getoutput(\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\")\n    pod_lines = pods_output.strip().split(\"\\n\")\n\n    with open(tmp_report_path, \"w\") as f:\n        f.write(f\"\u003chtml\u003e\u003cbody\u003e\u003ch1\u003e장애 복기 리포트 (keyword: `{keyword}`)\u003c/h1\u003e\\n\")\n\n        for line in pod_lines:\n            if not line.strip():\n                continue\n            try:\n                namespace, pod_name = line.split()\n            except ValueError:\n                continue\n\n            describe = subprocess.getoutput(f\"kubectl describe pod {pod_name} -n {namespace}\")\n            logs = subprocess.getoutput(f\"kubectl logs {pod_name} -n {namespace}\")\n\n            matched_describe = \"\\n\".join([l for l in describe.splitlines() if keyword in l])\n            matched_logs = \"\\n\".join([l for l in logs.splitlines() if keyword in l])\n\n            if matched_describe or matched_logs:\n                f.write(f\"\u003ch2\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\u003c/h2\u003e\\n\")\n                if matched_describe:\n                    f.write(f\"\u003ch3\u003eDescribe\u003c/h3\u003e\u003cpre\u003e{matched_describe}\u003c/pre\u003e\\n\")\n                if matched_logs:\n                    f.write(f\"\u003ch3\u003eLogs\u003c/h3\u003e\u003cpre\u003e{matched_logs}\u003c/pre\u003e\\n\")\n\n            if namespace not in seen_namespaces:\n                events = subprocess.getoutput(f\"kubectl get events -n {namespace}\")\n                matched_events = \"\\n\".join([l for l in events.splitlines() if keyword in l])\n                if matched_events:\n                    namespace_events[namespace] = matched_events\n                seen_namespaces.add(namespace)\n\n        # Write filtered events grouped by namespace\n        for ns, events in namespace_events.items():\n            f.write(f\"\u003ch3\u003eEvents in Namespace: `{ns}`\u003c/h3\u003e\u003cpre\u003e{events}\u003c/pre\u003e\\n\")\n\n        f.write(\"\u003c/body\u003e\u003c/html\u003e\")\n\n    subprocess.run([\"cp\", tmp_report_path, output_report.path], check=True)\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"128"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","01d67935-b0f8-4b00-b0de-e304235cd881","--dag_execution_id","128","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-scgtw/2025/05/08/pod-log-to-github-scgtw-system-container-driver-420083653"}}}

Logs (filtered)

time="2025-05-08T05:51:06.324Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-scgtw-system-container-impl-2117547052 (Namespace: kubeflow-user-example-com)

Describe (filtered)

Status:       Failed
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"01d67935-b0f8-4b00-b0de-e304235cd881\",\"--execution_id\",\"130\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"20\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/01d67935-b0f8-4b00-b0de-e304235cd881/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-scgtw/2025/05/08/pod-log-to-github-scgtw-system-container-impl-2117547052"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"01d67935-b0f8-4b00-b0de-e304235cd881\",\"--execution_id\",\"130\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"20\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/01d67935-b0f8-4b00-b0de-e304235cd881/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"01d67935-b0f8-4b00-b0de-e304235cd881\",\"--execution_id\",\"130\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"20\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/01d67935-b0f8-4b00-b0de-e304235cd881/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-scgtw/2025/05/08/pod-log-to-github-scgtw-system-container-impl-2117547052"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"01d67935-b0f8-4b00-b0de-e304235cd881\",\"--execution_id\",\"130\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"20\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/01d67935-b0f8-4b00-b0de-e304235cd881/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"01d67935-b0f8-4b00-b0de-e304235cd881\",\"--execution_id\",\"130\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"20\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/01d67935-b0f8-4b00-b0de-e304235cd881/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-scgtw/2025/05/08/pod-log-to-github-scgtw-system-container-impl-2117547052"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"01d67935-b0f8-4b00-b0de-e304235cd881\",\"--execution_id\",\"130\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"20\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/01d67935-b0f8-4b00-b0de-e304235cd881/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
              raise FileNotFoundError(f"{report_path} does not exist")
      Reason:       Error
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"01d67935-b0f8-4b00-b0de-e304235cd881\",\"--execution_id\",\"130\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"20\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/01d67935-b0f8-4b00-b0de-e304235cd881/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-scgtw/2025/05/08/pod-log-to-github-scgtw-system-container-impl-2117547052"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"01d67935-b0f8-4b00-b0de-e304235cd881\",\"--execution_id\",\"130\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"20\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/01d67935-b0f8-4b00-b0de-e304235cd881/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}

Logs (filtered)

remote: error: GH013: Repository rule violations found for refs/heads/report.        
error: failed to push some refs to 'https://github.com/Rosa1026/Kubeflow.git'
Traceback (most recent call last):
F0508 05:52:59.034433      22 main.go:49] failed to execute component: exit status 1
time="2025-05-08T05:52:59.563Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T05:53:00.505Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-scgtw-system-container-impl-2410495459 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"01d67935-b0f8-4b00-b0de-e304235cd881\",\"--execution_id\",\"129\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/01d67935-b0f8-4b00-b0de-e304235cd881/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-scgtw/2025/05/08/pod-log-to-github-scgtw-system-container-impl-2410495459"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"01d67935-b0f8-4b00-b0de-e304235cd881\",\"--execution_id\",\"129\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/01d67935-b0f8-4b00-b0de-e304235cd881/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"01d67935-b0f8-4b00-b0de-e304235cd881\",\"--execution_id\",\"129\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/01d67935-b0f8-4b00-b0de-e304235cd881/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-scgtw/2025/05/08/pod-log-to-github-scgtw-system-container-impl-2410495459"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"01d67935-b0f8-4b00-b0de-e304235cd881\",\"--execution_id\",\"129\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/01d67935-b0f8-4b00-b0de-e304235cd881/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"01d67935-b0f8-4b00-b0de-e304235cd881\",\"--execution_id\",\"129\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/01d67935-b0f8-4b00-b0de-e304235cd881/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-scgtw/2025/05/08/pod-log-to-github-scgtw-system-container-impl-2410495459"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"01d67935-b0f8-4b00-b0de-e304235cd881\",\"--execution_id\",\"129\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/01d67935-b0f8-4b00-b0de-e304235cd881/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
                  except ValueError:
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"01d67935-b0f8-4b00-b0de-e304235cd881\",\"--execution_id\",\"129\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/01d67935-b0f8-4b00-b0de-e304235cd881/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-scgtw/2025/05/08/pod-log-to-github-scgtw-system-container-impl-2410495459"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"01d67935-b0f8-4b00-b0de-e304235cd881\",\"--execution_id\",\"129\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/01d67935-b0f8-4b00-b0de-e304235cd881/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}

Logs (filtered)

time="2025-05-08T05:52:25.618Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T05:52:26.551Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-scgtw-system-dag-driver-49212381 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T05:50:57.241Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-sqv6q-system-container-driver-2710076593 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"84"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","480c9d7a-bc52-4ee1-85a0-58ef97c7f10e","--dag_execution_id","84","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-sqv6q/2025/05/08/pod-log-to-github-sqv6q-system-container-driver-2710076593"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"84"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","480c9d7a-bc52-4ee1-85a0-58ef97c7f10e","--dag_execution_id","84","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-sqv6q/2025/05/08/pod-log-to-github-sqv6q-system-container-driver-2710076593"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str = \"main\"\n):\n    import os\n    import subprocess\n\n\n    report_path = report_file.path\n    print(\"Checking if report exists at:\", report_path)\n    if not os.path.exists(report_path):\n        raise FileNotFoundError(f\"{report_path} does not exist\")\n\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    subprocess.run([\"cp\", report_path, os.path.join(repo_dir, \"report.md\")], check=True)\n\n    # Git user config\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", \"Add report\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(\"Report committed and pushed to GitHub.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"84"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","480c9d7a-bc52-4ee1-85a0-58ef97c7f10e","--dag_execution_id","84","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-all-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-all-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-sqv6q/2025/05/08/pod-log-to-github-sqv6q-system-container-driver-2710076593"}}}

Logs (filtered)

time="2025-05-08T00:35:19.823Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-sqv6q-system-container-driver-5786250 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T00:34:10.145Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-sqv6q-system-container-impl-2441006692 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T00:35:06.455Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T00:35:07.397Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-sqv6q-system-container-impl-41613183 (Namespace: kubeflow-user-example-com)

Describe (filtered)

Status:       Failed
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"480c9d7a-bc52-4ee1-85a0-58ef97c7f10e\",\"--execution_id\",\"86\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"6\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/480c9d7a-bc52-4ee1-85a0-58ef97c7f10e/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-sqv6q/2025/05/08/pod-log-to-github-sqv6q-system-container-impl-41613183"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"480c9d7a-bc52-4ee1-85a0-58ef97c7f10e\",\"--execution_id\",\"86\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"6\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/480c9d7a-bc52-4ee1-85a0-58ef97c7f10e/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"480c9d7a-bc52-4ee1-85a0-58ef97c7f10e\",\"--execution_id\",\"86\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"6\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/480c9d7a-bc52-4ee1-85a0-58ef97c7f10e/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-sqv6q/2025/05/08/pod-log-to-github-sqv6q-system-container-impl-41613183"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"480c9d7a-bc52-4ee1-85a0-58ef97c7f10e\",\"--execution_id\",\"86\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"6\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/480c9d7a-bc52-4ee1-85a0-58ef97c7f10e/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"480c9d7a-bc52-4ee1-85a0-58ef97c7f10e\",\"--execution_id\",\"86\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"6\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/480c9d7a-bc52-4ee1-85a0-58ef97c7f10e/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-sqv6q/2025/05/08/pod-log-to-github-sqv6q-system-container-impl-41613183"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"480c9d7a-bc52-4ee1-85a0-58ef97c7f10e\",\"--execution_id\",\"86\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"6\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/480c9d7a-bc52-4ee1-85a0-58ef97c7f10e/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
              raise FileNotFoundError(f"{report_path} does not exist")
      Reason:       Error
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"480c9d7a-bc52-4ee1-85a0-58ef97c7f10e\",\"--execution_id\",\"86\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"6\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/480c9d7a-bc52-4ee1-85a0-58ef97c7f10e/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-sqv6q/2025/05/08/pod-log-to-github-sqv6q-system-container-impl-41613183"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"480c9d7a-bc52-4ee1-85a0-58ef97c7f10e\",\"--execution_id\",\"86\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"6\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/480c9d7a-bc52-4ee1-85a0-58ef97c7f10e/collect-all-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}

Logs (filtered)

remote: error: GH013: Repository rule violations found for refs/heads/main.        
error: failed to push some refs to 'https://github.com/Rosa1026/Kubeflow.git'
Traceback (most recent call last):
F0508 00:35:39.921601      24 main.go:49] failed to execute component: exit status 1
time="2025-05-08T00:35:40.046Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T00:35:40.989Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-sqv6q-system-dag-driver-3865249824 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T00:34:01.119Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-ssnsq-system-container-driver-4260920965 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T08:08:56.431Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-ssnsq-system-container-driver-948089400 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"61"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","3e83b1f6-e285-4031-9108-07a6049eac87","--dag_execution_id","61","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-ssnsq/2025/05/07/pod-log-to-github-ssnsq-system-container-driver-948089400"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"61"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","3e83b1f6-e285-4031-9108-07a6049eac87","--dag_execution_id","61","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-ssnsq/2025/05/07/pod-log-to-github-ssnsq-system-container-driver-948089400"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str = \"main\"\n):\n    import os\n    import subprocess\n\n\n    report_path = report_file.path\n    print(\"Checking if report exists at:\", report_path)\n    if not os.path.exists(report_path):\n        raise FileNotFoundError(f\"{report_path} does not exist\")\n\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    subprocess.run([\"cp\", report_path, os.path.join(repo_dir, \"report.md\")], check=True)\n\n    # Git user config\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", \"Add report\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(\"Report committed and pushed to GitHub.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"61"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","3e83b1f6-e285-4031-9108-07a6049eac87","--dag_execution_id","61","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-ssnsq/2025/05/07/pod-log-to-github-ssnsq-system-container-driver-948089400"}}}

Logs (filtered)

time="2025-05-07T08:09:34.825Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-ssnsq-system-container-impl-1620770958 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"3e83b1f6-e285-4031-9108-07a6049eac87\",\"--execution_id\",\"63\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"4\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/3e83b1f6-e285-4031-9108-07a6049eac87/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-ssnsq/2025/05/07/pod-log-to-github-ssnsq-system-container-impl-1620770958"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"3e83b1f6-e285-4031-9108-07a6049eac87\",\"--execution_id\",\"63\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"4\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/3e83b1f6-e285-4031-9108-07a6049eac87/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"3e83b1f6-e285-4031-9108-07a6049eac87\",\"--execution_id\",\"63\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"4\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/3e83b1f6-e285-4031-9108-07a6049eac87/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-ssnsq/2025/05/07/pod-log-to-github-ssnsq-system-container-impl-1620770958"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"3e83b1f6-e285-4031-9108-07a6049eac87\",\"--execution_id\",\"63\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"4\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/3e83b1f6-e285-4031-9108-07a6049eac87/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"3e83b1f6-e285-4031-9108-07a6049eac87\",\"--execution_id\",\"63\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"4\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/3e83b1f6-e285-4031-9108-07a6049eac87/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-ssnsq/2025/05/07/pod-log-to-github-ssnsq-system-container-impl-1620770958"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"3e83b1f6-e285-4031-9108-07a6049eac87\",\"--execution_id\",\"63\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"4\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/3e83b1f6-e285-4031-9108-07a6049eac87/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
              raise FileNotFoundError(f"{report_path} does not exist")
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"3e83b1f6-e285-4031-9108-07a6049eac87\",\"--execution_id\",\"63\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"4\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/3e83b1f6-e285-4031-9108-07a6049eac87/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-ssnsq/2025/05/07/pod-log-to-github-ssnsq-system-container-impl-1620770958"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"3e83b1f6-e285-4031-9108-07a6049eac87\",\"--execution_id\",\"63\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"4\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/3e83b1f6-e285-4031-9108-07a6049eac87/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}

Logs (filtered)

time="2025-05-07T08:09:52.925Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-07T08:09:53.866Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-ssnsq-system-container-impl-3899670179 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T08:09:20.635Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-07T08:09:21.564Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-ssnsq-system-dag-driver-1298226427 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T08:08:46.326Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-t8pxp-system-container-driver-3201759859 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"event_keywords\":{\"defaultValue\":\"crash,unhealthy\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"log_keywords\":{\"defaultValue\":\"crash,error,fail\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"inputs\":{\"parameters\":{\"event_keywords\":{\"componentInputParameter\":\"event_keywords\"},\"log_keywords\":{\"componentInputParameter\":\"log_keywords\"}}},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(\\n    output_report: Output[Dataset],\\n    log_keywords: str = \\\"crash,error,fail\\\",\\n    event_keywords: str = \\\"crash,unhealthy\\\"\\n):\\n    import subprocess\\n    import shutil\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.md\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    def filter_lines(text, keywords):\\n        return [line for line in text.splitlines() if any(k.lower() in line.lower() for k in keywords)]\\n\\n    log_kw_list = [k.strip() for k in log_keywords.split(\\\",\\\") if k.strip()]\\n    event_kw_list = [k.strip() for k in event_keywords.split(\\\",\\\") if k.strip()]\\n\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"# 장애 복기 리포트\\\\n\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join(filter_lines(describe, log_kw_list))\\n            matched_logs = \\\"\\\\n\\\".join(filter_lines(logs, log_kw_list))\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\\\n\\\")\\n                if matched_describe:\\n                    f.write(\\\"### describe\\\\n```text\\\\n\\\" + matched_describe + \\\"\\\\n```\\\\n\\\")\\n                if matched_logs:\\n                    f.write(\\\"### logs\\\\n```text\\\\n\\\" + matched_logs + \\\"\\\\n```\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join(filter_lines(events, event_kw_list))\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\\\n## Events in Namespace: `{ns}`\\\\n```text\\\\n{events}\\\\n```\\\\n\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"105"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","b0b52bad-4143-40c8-b509-602b3b0542d6","--dag_execution_id","105","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"event_keywords\":{\"defaultValue\":\"crash,unhealthy\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"log_keywords\":{\"defaultValue\":\"crash,error,fail\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"inputs\":{\"parameters\":{\"event_keywords\":{\"componentInputParameter\":\"event_keywords\"},\"log_keywords\":{\"componentInputParameter\":\"log_keywords\"}}},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(\\n    output_report: Output[Dataset],\\n    log_keywords: str = \\\"crash,error,fail\\\",\\n    event_keywords: str = \\\"crash,unhealthy\\\"\\n):\\n    import subprocess\\n    import shutil\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.md\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    def filter_lines(text, keywords):\\n        return [line for line in text.splitlines() if any(k.lower() in line.lower() for k in keywords)]\\n\\n    log_kw_list = [k.strip() for k in log_keywords.split(\\\",\\\") if k.strip()]\\n    event_kw_list = [k.strip() for k in event_keywords.split(\\\",\\\") if k.strip()]\\n\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"# 장애 복기 리포트\\\\n\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join(filter_lines(describe, log_kw_list))\\n            matched_logs = \\\"\\\\n\\\".join(filter_lines(logs, log_kw_list))\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\\\n\\\")\\n                if matched_describe:\\n                    f.write(\\\"### describe\\\\n```text\\\\n\\\" + matched_describe + \\\"\\\\n```\\\\n\\\")\\n                if matched_logs:\\n                    f.write(\\\"### logs\\\\n```text\\\\n\\\" + matched_logs + \\\"\\\\n```\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join(filter_lines(events, event_kw_list))\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\\\n## Events in Namespace: `{ns}`\\\\n```text\\\\n{events}\\\\n```\\\\n\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-t8pxp/2025/05/08/pod-log-to-github-t8pxp-system-container-driver-3201759859"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"event_keywords\":{\"defaultValue\":\"crash,unhealthy\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"log_keywords\":{\"defaultValue\":\"crash,error,fail\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"inputs\":{\"parameters\":{\"event_keywords\":{\"componentInputParameter\":\"event_keywords\"},\"log_keywords\":{\"componentInputParameter\":\"log_keywords\"}}},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(\\n    output_report: Output[Dataset],\\n    log_keywords: str = \\\"crash,error,fail\\\",\\n    event_keywords: str = \\\"crash,unhealthy\\\"\\n):\\n    import subprocess\\n    import shutil\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.md\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    def filter_lines(text, keywords):\\n        return [line for line in text.splitlines() if any(k.lower() in line.lower() for k in keywords)]\\n\\n    log_kw_list = [k.strip() for k in log_keywords.split(\\\",\\\") if k.strip()]\\n    event_kw_list = [k.strip() for k in event_keywords.split(\\\",\\\") if k.strip()]\\n\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"# 장애 복기 리포트\\\\n\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join(filter_lines(describe, log_kw_list))\\n            matched_logs = \\\"\\\\n\\\".join(filter_lines(logs, log_kw_list))\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\\\n\\\")\\n                if matched_describe:\\n                    f.write(\\\"### describe\\\\n```text\\\\n\\\" + matched_describe + \\\"\\\\n```\\\\n\\\")\\n                if matched_logs:\\n                    f.write(\\\"### logs\\\\n```text\\\\n\\\" + matched_logs + \\\"\\\\n```\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join(filter_lines(events, event_kw_list))\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\\\n## Events in Namespace: `{ns}`\\\\n```text\\\\n{events}\\\\n```\\\\n\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"105"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","b0b52bad-4143-40c8-b509-602b3b0542d6","--dag_execution_id","105","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"event_keywords\":{\"defaultValue\":\"crash,unhealthy\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"log_keywords\":{\"defaultValue\":\"crash,error,fail\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"inputs\":{\"parameters\":{\"event_keywords\":{\"componentInputParameter\":\"event_keywords\"},\"log_keywords\":{\"componentInputParameter\":\"log_keywords\"}}},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(\\n    output_report: Output[Dataset],\\n    log_keywords: str = \\\"crash,error,fail\\\",\\n    event_keywords: str = \\\"crash,unhealthy\\\"\\n):\\n    import subprocess\\n    import shutil\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.md\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    def filter_lines(text, keywords):\\n        return [line for line in text.splitlines() if any(k.lower() in line.lower() for k in keywords)]\\n\\n    log_kw_list = [k.strip() for k in log_keywords.split(\\\",\\\") if k.strip()]\\n    event_kw_list = [k.strip() for k in event_keywords.split(\\\",\\\") if k.strip()]\\n\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"# 장애 복기 리포트\\\\n\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join(filter_lines(describe, log_kw_list))\\n            matched_logs = \\\"\\\\n\\\".join(filter_lines(logs, log_kw_list))\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\\\n\\\")\\n                if matched_describe:\\n                    f.write(\\\"### describe\\\\n```text\\\\n\\\" + matched_describe + \\\"\\\\n```\\\\n\\\")\\n                if matched_logs:\\n                    f.write(\\\"### logs\\\\n```text\\\\n\\\" + matched_logs + \\\"\\\\n```\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join(filter_lines(events, event_kw_list))\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\\\n## Events in Namespace: `{ns}`\\\\n```text\\\\n{events}\\\\n```\\\\n\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-t8pxp/2025/05/08/pod-log-to-github-t8pxp-system-container-driver-3201759859"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","collect_filtered_logs"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef collect_filtered_logs(\n    output_report: Output[Dataset],\n    log_keywords: str = \"crash,error,fail\",\n    event_keywords: str = \"crash,unhealthy\"\n):\n    import subprocess\n    import shutil\n    import os\n\n    tmp_report_path = \"/tmp/report.md\"\n    seen_namespaces = set()\n    namespace_events = {}\n\n    def filter_lines(text, keywords):\n        return [line for line in text.splitlines() if any(k.lower() in line.lower() for k in keywords)]\n\n    log_kw_list = [k.strip() for k in log_keywords.split(\",\") if k.strip()]\n    event_kw_list = [k.strip() for k in event_keywords.split(\",\") if k.strip()]\n\n    subprocess.run([\"apt-get\", \"update\", \"-y\"], check=True)\n    subprocess.run([\n        \"curl\", \"-LO\", \"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\"\n    ], check=True)\n    subprocess.run([\"chmod\", \"+x\", \"kubectl\"], check=True)\n    subprocess.run([\"mv\", \"kubectl\", \"/usr/local/bin/\"], check=True)\n\n    pods_output = subprocess.getoutput(\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\")\n    pod_lines = pods_output.strip().split(\"\\n\")\n\n    with open(tmp_report_path, \"w\") as f:\n        f.write(f\"# 장애 복기 리포트\\n\\n\")\n\n        for line in pod_lines:\n            if not line.strip():\n                continue\n            try:\n                namespace, pod_name = line.split()\n            except ValueError:\n                continue\n\n            describe = subprocess.getoutput(f\"kubectl describe pod {pod_name} -n {namespace}\")\n            logs = subprocess.getoutput(f\"kubectl logs {pod_name} -n {namespace}\")\n\n            matched_describe = \"\\n\".join(filter_lines(describe, log_kw_list))\n            matched_logs = \"\\n\".join(filter_lines(logs, log_kw_list))\n\n            if matched_describe or matched_logs:\n                f.write(f\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\n\")\n                if matched_describe:\n                    f.write(\"### describe\\n```text\\n\" + matched_describe + \"\\n```\\n\")\n                if matched_logs:\n                    f.write(\"### logs\\n```text\\n\" + matched_logs + \"\\n```\\n\")\n\n            if namespace not in seen_namespaces:\n                events = subprocess.getoutput(f\"kubectl get events -n {namespace}\")\n                matched_events = \"\\n\".join(filter_lines(events, event_kw_list))\n                if matched_events:\n                    namespace_events[namespace] = matched_events\n                seen_namespaces.add(namespace)\n\n        for ns, events in namespace_events.items():\n            f.write(f\"\\n## Events in Namespace: `{ns}`\\n```text\\n{events}\\n```\\n\")\n\n    subprocess.run([\"cp\", tmp_report_path, output_report.path], check=True)\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"event_keywords\":{\"defaultValue\":\"crash,unhealthy\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"log_keywords\":{\"defaultValue\":\"crash,error,fail\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"inputs\":{\"parameters\":{\"event_keywords\":{\"componentInputParameter\":\"event_keywords\"},\"log_keywords\":{\"componentInputParameter\":\"log_keywords\"}}},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(\\n    output_report: Output[Dataset],\\n    log_keywords: str = \\\"crash,error,fail\\\",\\n    event_keywords: str = \\\"crash,unhealthy\\\"\\n):\\n    import subprocess\\n    import shutil\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.md\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    def filter_lines(text, keywords):\\n        return [line for line in text.splitlines() if any(k.lower() in line.lower() for k in keywords)]\\n\\n    log_kw_list = [k.strip() for k in log_keywords.split(\\\",\\\") if k.strip()]\\n    event_kw_list = [k.strip() for k in event_keywords.split(\\\",\\\") if k.strip()]\\n\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"# 장애 복기 리포트\\\\n\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join(filter_lines(describe, log_kw_list))\\n            matched_logs = \\\"\\\\n\\\".join(filter_lines(logs, log_kw_list))\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\\\n\\\")\\n                if matched_describe:\\n                    f.write(\\\"### describe\\\\n```text\\\\n\\\" + matched_describe + \\\"\\\\n```\\\\n\\\")\\n                if matched_logs:\\n                    f.write(\\\"### logs\\\\n```text\\\\n\\\" + matched_logs + \\\"\\\\n```\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join(filter_lines(events, event_kw_list))\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\\\n## Events in Namespace: `{ns}`\\\\n```text\\\\n{events}\\\\n```\\\\n\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"105"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","b0b52bad-4143-40c8-b509-602b3b0542d6","--dag_execution_id","105","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"event_keywords\":{\"defaultValue\":\"crash,unhealthy\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"log_keywords\":{\"defaultValue\":\"crash,error,fail\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"inputs\":{\"parameters\":{\"event_keywords\":{\"componentInputParameter\":\"event_keywords\"},\"log_keywords\":{\"componentInputParameter\":\"log_keywords\"}}},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(\\n    output_report: Output[Dataset],\\n    log_keywords: str = \\\"crash,error,fail\\\",\\n    event_keywords: str = \\\"crash,unhealthy\\\"\\n):\\n    import subprocess\\n    import shutil\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.md\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    def filter_lines(text, keywords):\\n        return [line for line in text.splitlines() if any(k.lower() in line.lower() for k in keywords)]\\n\\n    log_kw_list = [k.strip() for k in log_keywords.split(\\\",\\\") if k.strip()]\\n    event_kw_list = [k.strip() for k in event_keywords.split(\\\",\\\") if k.strip()]\\n\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"# 장애 복기 리포트\\\\n\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join(filter_lines(describe, log_kw_list))\\n            matched_logs = \\\"\\\\n\\\".join(filter_lines(logs, log_kw_list))\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\\\n\\\")\\n                if matched_describe:\\n                    f.write(\\\"### describe\\\\n```text\\\\n\\\" + matched_describe + \\\"\\\\n```\\\\n\\\")\\n                if matched_logs:\\n                    f.write(\\\"### logs\\\\n```text\\\\n\\\" + matched_logs + \\\"\\\\n```\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join(filter_lines(events, event_kw_list))\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\\\n## Events in Namespace: `{ns}`\\\\n```text\\\\n{events}\\\\n```\\\\n\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-t8pxp/2025/05/08/pod-log-to-github-t8pxp-system-container-driver-3201759859"}}}

Logs (filtered)

        "defaultValue": "crash,error,fail",
    "\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef collect_filtered_logs(\n    output_report: Output[Dataset],\n    log_keywords: str = \"crash,error,fail\",\n    event_keywords: str = \"crash,unhealthy\"\n):\n    import subprocess\n    import shutil\n    import os\n\n    tmp_report_path = \"/tmp/report.md\"\n    seen_namespaces = set()\n    namespace_events = {}\n\n    def filter_lines(text, keywords):\n        return [line for line in text.splitlines() if any(k.lower() in line.lower() for k in keywords)]\n\n    log_kw_list = [k.strip() for k in log_keywords.split(\",\") if k.strip()]\n    event_kw_list = [k.strip() for k in event_keywords.split(\",\") if k.strip()]\n\n    subprocess.run([\"apt-get\", \"update\", \"-y\"], check=True)\n    subprocess.run([\n        \"curl\", \"-LO\", \"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\"\n    ], check=True)\n    subprocess.run([\"chmod\", \"+x\", \"kubectl\"], check=True)\n    subprocess.run([\"mv\", \"kubectl\", \"/usr/local/bin/\"], check=True)\n\n    pods_output = subprocess.getoutput(\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\")\n    pod_lines = pods_output.strip().split(\"\\n\")\n\n    with open(tmp_report_path, \"w\") as f:\n        f.write(f\"# 장애 복기 리포트\\n\\n\")\n\n        for line in pod_lines:\n            if not line.strip():\n                continue\n            try:\n                namespace, pod_name = line.split()\n            except ValueError:\n                continue\n\n            describe = subprocess.getoutput(f\"kubectl describe pod {pod_name} -n {namespace}\")\n            logs = subprocess.getoutput(f\"kubectl logs {pod_name} -n {namespace}\")\n\n            matched_describe = \"\\n\".join(filter_lines(describe, log_kw_list))\n            matched_logs = \"\\n\".join(filter_lines(logs, log_kw_list))\n\n            if matched_describe or matched_logs:\n                f.write(f\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\n\")\n                if matched_describe:\n                    f.write(\"### describe\\n```text\\n\" + matched_describe + \"\\n```\\n\")\n                if matched_logs:\n                    f.write(\"### logs\\n```text\\n\" + matched_logs + \"\\n```\\n\")\n\n            if namespace not in seen_namespaces:\n                events = subprocess.getoutput(f\"kubectl get events -n {namespace}\")\n                matched_events = \"\\n\".join(filter_lines(events, event_kw_list))\n                if matched_events:\n                    namespace_events[namespace] = matched_events\n                seen_namespaces.add(namespace)\n\n        for ns, events in namespace_events.items():\n            f.write(f\"\\n## Events in Namespace: `{ns}`\\n```text\\n{events}\\n```\\n\")\n\n    subprocess.run([\"cp\", tmp_report_path, output_report.path], check=True)\n\n"
I0508 04:29:01.066996      14 driver.go:252] parent DAG: id:105  name:"run/b0b52bad-4143-40c8-b509-602b3b0542d6"  type_id:13  type:"system.DAGExecution"  last_known_state:RUNNING  custom_properties:{key:"display_name"  value:{string_value:""}}  custom_properties:{key:"inputs"  value:{struct_value:{fields:{key:"branch"  value:{string_value:"main"}}  fields:{key:"event_keywords"  value:{string_value:"crash,unhealthy"}}  fields:{key:"github_token"  value:{string_value:"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"}}  fields:{key:"log_keywords"  value:{string_value:"crash,error,fail"}}  fields:{key:"repo_url"  value:{string_value:"https://github.com/Rosa1026/Kubeflow.git"}}}}}  custom_properties:{key:"task_name"  value:{string_value:""}}  create_time_since_epoch:1746678530959  last_update_time_since_epoch:1746678530959
I0508 04:29:01.112877      14 driver.go:926] parent DAG input parameters: map[branch:string_value:"main" event_keywords:string_value:"crash,unhealthy" github_token:string_value:"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" log_keywords:string_value:"crash,error,fail" repo_url:string_value:"https://github.com/Rosa1026/Kubeflow.git"], artifacts: map[]
I0508 04:29:01.190342      14 driver.go:310] Created execution: id:106  type_id:14  type:"system.ContainerExecution"  last_known_state:RUNNING  custom_properties:{key:"display_name"  value:{string_value:"collect-filtered-logs"}}  custom_properties:{key:"image"  value:{string_value:""}}  custom_properties:{key:"inputs"  value:{struct_value:{fields:{key:"event_keywords"  value:{string_value:"crash,unhealthy"}}  fields:{key:"log_keywords"  value:{string_value:"crash,error,fail"}}}}}  custom_properties:{key:"namespace"  value:{string_value:""}}  custom_properties:{key:"parent_dag_id"  value:{int_value:105}}  custom_properties:{key:"pod_name"  value:{string_value:""}}  custom_properties:{key:"pod_uid"  value:{string_value:""}}  custom_properties:{key:"task_name"  value:{string_value:"collect-filtered-logs"}}  create_time_since_epoch:1746678541155  last_update_time_since_epoch:1746678541155
{"containers":[{"name":"main","image":"python:3.9","command":["/var/run/argo/argoexec","emissary","--","/kfp-launcher/launch","--pipeline_name","pod-log-to-github","--run_id","b0b52bad-4143-40c8-b509-602b3b0542d6","--execution_id","106","--executor_input","{\"inputs\":{\"parameterValues\":{\"event_keywords\":\"crash,unhealthy\", \"log_keywords\":\"crash,error,fail\"}}, \"outputs\":{\"artifacts\":{\"output_report\":{\"artifacts\":[{\"type\":{\"schemaTitle\":\"system.Dataset\", \"schemaVersion\":\"0.0.1\"}, \"uri\":\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b0b52bad-4143-40c8-b509-602b3b0542d6/collect-filtered-logs/output_report\"}]}}, \"outputFile\":\"/tmp/kfp_outputs/output_metadata.json\"}}","--component_spec","{\"inputDefinitions\":{\"parameters\":{\"event_keywords\":{\"parameterType\":\"STRING\", \"defaultValue\":\"crash,unhealthy\", \"isOptional\":true}, \"log_keywords\":{\"parameterType\":\"STRING\", \"defaultValue\":\"crash,error,fail\", \"isOptional\":true}}}, \"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\", \"schemaVersion\":\"0.0.1\"}}}}, \"executorLabel\":\"exec-collect-filtered-logs\"}","--pod_name","$(KFP_POD_NAME)","--pod_uid","$(KFP_POD_UID)","--mlmd_server_address","$(METADATA_GRPC_SERVICE_HOST)","--mlmd_server_port","$(METADATA_GRPC_SERVICE_PORT)","--"],"args":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef collect_filtered_logs(\n    output_report: Output[Dataset],\n    log_keywords: str = \"crash,error,fail\",\n    event_keywords: str = \"crash,unhealthy\"\n):\n    import subprocess\n    import shutil\n    import os\n\n    tmp_report_path = \"/tmp/report.md\"\n    seen_namespaces = set()\n    namespace_events = {}\n\n    def filter_lines(text, keywords):\n        return [line for line in text.splitlines() if any(k.lower() in line.lower() for k in keywords)]\n\n    log_kw_list = [k.strip() for k in log_keywords.split(\",\") if k.strip()]\n    event_kw_list = [k.strip() for k in event_keywords.split(\",\") if k.strip()]\n\n    subprocess.run([\"apt-get\", \"update\", \"-y\"], check=True)\n    subprocess.run([\n        \"curl\", \"-LO\", \"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\"\n    ], check=True)\n    subprocess.run([\"chmod\", \"+x\", \"kubectl\"], check=True)\n    subprocess.run([\"mv\", \"kubectl\", \"/usr/local/bin/\"], check=True)\n\n    pods_output = subprocess.getoutput(\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\")\n    pod_lines = pods_output.strip().split(\"\\n\")\n\n    with open(tmp_report_path, \"w\") as f:\n        f.write(f\"# 장애 복기 리포트\\n\\n\")\n\n        for line in pod_lines:\n            if not line.strip():\n                continue\n            try:\n                namespace, pod_name = line.split()\n            except ValueError:\n                continue\n\n            describe = subprocess.getoutput(f\"kubectl describe pod {pod_name} -n {namespace}\")\n            logs = subprocess.getoutput(f\"kubectl logs {pod_name} -n {namespace}\")\n\n            matched_describe = \"\\n\".join(filter_lines(describe, log_kw_list))\n            matched_logs = \"\\n\".join(filter_lines(logs, log_kw_list))\n\n            if matched_describe or matched_logs:\n                f.write(f\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\n\")\n                if matched_describe:\n                    f.write(\"### describe\\n```text\\n\" + matched_describe + \"\\n```\\n\")\n                if matched_logs:\n                    f.write(\"### logs\\n```text\\n\" + matched_logs + \"\\n```\\n\")\n\n            if namespace not in seen_namespaces:\n                events = subprocess.getoutput(f\"kubectl get events -n {namespace}\")\n                matched_events = \"\\n\".join(filter_lines(events, event_kw_list))\n                if matched_events:\n                    namespace_events[namespace] = matched_events\n                seen_namespaces.add(namespace)\n\n        for ns, events in namespace_events.items():\n            f.write(f\"\\n## Events in Namespace: `{ns}`\\n```text\\n{events}\\n```\\n\")\n\n    subprocess.run([\"cp\", tmp_report_path, output_report.path], check=True)\n\n","--executor_input","{{$}}","--function_to_execute","collect_filtered_logs"],"resources":{}}]}
      "log_keywords": "crash,error,fail"
time="2025-05-08T04:29:01.753Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-t8pxp-system-container-driver-85999812 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"105"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","b0b52bad-4143-40c8-b509-602b3b0542d6","--dag_execution_id","105","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-t8pxp/2025/05/08/pod-log-to-github-t8pxp-system-container-driver-85999812"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"105"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","b0b52bad-4143-40c8-b509-602b3b0542d6","--dag_execution_id","105","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-t8pxp/2025/05/08/pod-log-to-github-t8pxp-system-container-driver-85999812"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str = \"main\"\n):\n    import os\n    import subprocess\n\n    report_path = report_file.path\n    print(\"Checking if report exists at:\", report_path)\n    if not os.path.exists(report_path):\n        raise FileNotFoundError(f\"{report_path} does not exist\")\n\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    subprocess.run([\"cp\", report_path, os.path.join(repo_dir, \"report.md\")], check=True)\n\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", \"Add report\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(\"Report committed and pushed to GitHub.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"105"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","b0b52bad-4143-40c8-b509-602b3b0542d6","--dag_execution_id","105","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-t8pxp/2025/05/08/pod-log-to-github-t8pxp-system-container-driver-85999812"}}}

Logs (filtered)

I0508 04:30:30.895311      16 driver.go:252] parent DAG: id:105  name:"run/b0b52bad-4143-40c8-b509-602b3b0542d6"  type_id:13  type:"system.DAGExecution"  last_known_state:RUNNING  custom_properties:{key:"display_name"  value:{string_value:""}}  custom_properties:{key:"inputs"  value:{struct_value:{fields:{key:"branch"  value:{string_value:"main"}}  fields:{key:"event_keywords"  value:{string_value:"crash,unhealthy"}}  fields:{key:"github_token"  value:{string_value:"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"}}  fields:{key:"log_keywords"  value:{string_value:"crash,error,fail"}}  fields:{key:"repo_url"  value:{string_value:"https://github.com/Rosa1026/Kubeflow.git"}}}}}  custom_properties:{key:"task_name"  value:{string_value:""}}  create_time_since_epoch:1746678530959  last_update_time_since_epoch:1746678530959
I0508 04:30:30.940270      16 driver.go:926] parent DAG input parameters: map[branch:string_value:"main" event_keywords:string_value:"crash,unhealthy" github_token:string_value:"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" log_keywords:string_value:"crash,error,fail" repo_url:string_value:"https://github.com/Rosa1026/Kubeflow.git"], artifacts: map[]
time="2025-05-08T04:30:31.630Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-t8pxp-system-container-impl-2090937994 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b0b52bad-4143-40c8-b509-602b3b0542d6\",\"--execution_id\",\"107\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"13\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b0b52bad-4143-40c8-b509-602b3b0542d6/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-t8pxp/2025/05/08/pod-log-to-github-t8pxp-system-container-impl-2090937994"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b0b52bad-4143-40c8-b509-602b3b0542d6\",\"--execution_id\",\"107\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"13\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b0b52bad-4143-40c8-b509-602b3b0542d6/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b0b52bad-4143-40c8-b509-602b3b0542d6\",\"--execution_id\",\"107\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"13\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b0b52bad-4143-40c8-b509-602b3b0542d6/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-t8pxp/2025/05/08/pod-log-to-github-t8pxp-system-container-impl-2090937994"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b0b52bad-4143-40c8-b509-602b3b0542d6\",\"--execution_id\",\"107\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"13\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b0b52bad-4143-40c8-b509-602b3b0542d6/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b0b52bad-4143-40c8-b509-602b3b0542d6\",\"--execution_id\",\"107\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"13\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b0b52bad-4143-40c8-b509-602b3b0542d6/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-t8pxp/2025/05/08/pod-log-to-github-t8pxp-system-container-impl-2090937994"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b0b52bad-4143-40c8-b509-602b3b0542d6\",\"--execution_id\",\"107\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"13\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b0b52bad-4143-40c8-b509-602b3b0542d6/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
              raise FileNotFoundError(f"{report_path} does not exist")
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b0b52bad-4143-40c8-b509-602b3b0542d6\",\"--execution_id\",\"107\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"13\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b0b52bad-4143-40c8-b509-602b3b0542d6/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-t8pxp/2025/05/08/pod-log-to-github-t8pxp-system-container-impl-2090937994"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b0b52bad-4143-40c8-b509-602b3b0542d6\",\"--execution_id\",\"107\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"13\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b0b52bad-4143-40c8-b509-602b3b0542d6/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}

Logs (filtered)

time="2025-05-08T04:30:52.876Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T04:30:53.817Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-t8pxp-system-container-impl-418163285 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b0b52bad-4143-40c8-b509-602b3b0542d6\",\"--execution_id\",\"106\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"parameterValues\\\":{\\\"event_keywords\\\":\\\"crash,unhealthy\\\", \\\"log_keywords\\\":\\\"crash,error,fail\\\"}}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b0b52bad-4143-40c8-b509-602b3b0542d6/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"event_keywords\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash,unhealthy\\\", \\\"isOptional\\\":true}, \\\"log_keywords\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash,error,fail\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(\\n    output_report: Output[Dataset],\\n    log_keywords: str = \\\"crash,error,fail\\\",\\n    event_keywords: str = \\\"crash,unhealthy\\\"\\n):\\n    import subprocess\\n    import shutil\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.md\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    def filter_lines(text, keywords):\\n        return [line for line in text.splitlines() if any(k.lower() in line.lower() for k in keywords)]\\n\\n    log_kw_list = [k.strip() for k in log_keywords.split(\\\",\\\") if k.strip()]\\n    event_kw_list = [k.strip() for k in event_keywords.split(\\\",\\\") if k.strip()]\\n\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"# 장애 복기 리포트\\\\n\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join(filter_lines(describe, log_kw_list))\\n            matched_logs = \\\"\\\\n\\\".join(filter_lines(logs, log_kw_list))\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\\\n\\\")\\n                if matched_describe:\\n                    f.write(\\\"### describe\\\\n```text\\\\n\\\" + matched_describe + \\\"\\\\n```\\\\n\\\")\\n                if matched_logs:\\n                    f.write(\\\"### logs\\\\n```text\\\\n\\\" + matched_logs + \\\"\\\\n```\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join(filter_lines(events, event_kw_list))\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\\\n## Events in Namespace: `{ns}`\\\\n```text\\\\n{events}\\\\n```\\\\n\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-t8pxp/2025/05/08/pod-log-to-github-t8pxp-system-container-impl-418163285"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b0b52bad-4143-40c8-b509-602b3b0542d6\",\"--execution_id\",\"106\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"parameterValues\\\":{\\\"event_keywords\\\":\\\"crash,unhealthy\\\", \\\"log_keywords\\\":\\\"crash,error,fail\\\"}}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b0b52bad-4143-40c8-b509-602b3b0542d6/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"event_keywords\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash,unhealthy\\\", \\\"isOptional\\\":true}, \\\"log_keywords\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash,error,fail\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(\\n    output_report: Output[Dataset],\\n    log_keywords: str = \\\"crash,error,fail\\\",\\n    event_keywords: str = \\\"crash,unhealthy\\\"\\n):\\n    import subprocess\\n    import shutil\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.md\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    def filter_lines(text, keywords):\\n        return [line for line in text.splitlines() if any(k.lower() in line.lower() for k in keywords)]\\n\\n    log_kw_list = [k.strip() for k in log_keywords.split(\\\",\\\") if k.strip()]\\n    event_kw_list = [k.strip() for k in event_keywords.split(\\\",\\\") if k.strip()]\\n\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"# 장애 복기 리포트\\\\n\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join(filter_lines(describe, log_kw_list))\\n            matched_logs = \\\"\\\\n\\\".join(filter_lines(logs, log_kw_list))\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\\\n\\\")\\n                if matched_describe:\\n                    f.write(\\\"### describe\\\\n```text\\\\n\\\" + matched_describe + \\\"\\\\n```\\\\n\\\")\\n                if matched_logs:\\n                    f.write(\\\"### logs\\\\n```text\\\\n\\\" + matched_logs + \\\"\\\\n```\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join(filter_lines(events, event_kw_list))\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\\\n## Events in Namespace: `{ns}`\\\\n```text\\\\n{events}\\\\n```\\\\n\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b0b52bad-4143-40c8-b509-602b3b0542d6\",\"--execution_id\",\"106\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"parameterValues\\\":{\\\"event_keywords\\\":\\\"crash,unhealthy\\\", \\\"log_keywords\\\":\\\"crash,error,fail\\\"}}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b0b52bad-4143-40c8-b509-602b3b0542d6/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"event_keywords\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash,unhealthy\\\", \\\"isOptional\\\":true}, \\\"log_keywords\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash,error,fail\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(\\n    output_report: Output[Dataset],\\n    log_keywords: str = \\\"crash,error,fail\\\",\\n    event_keywords: str = \\\"crash,unhealthy\\\"\\n):\\n    import subprocess\\n    import shutil\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.md\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    def filter_lines(text, keywords):\\n        return [line for line in text.splitlines() if any(k.lower() in line.lower() for k in keywords)]\\n\\n    log_kw_list = [k.strip() for k in log_keywords.split(\\\",\\\") if k.strip()]\\n    event_kw_list = [k.strip() for k in event_keywords.split(\\\",\\\") if k.strip()]\\n\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"# 장애 복기 리포트\\\\n\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join(filter_lines(describe, log_kw_list))\\n            matched_logs = \\\"\\\\n\\\".join(filter_lines(logs, log_kw_list))\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\\\n\\\")\\n                if matched_describe:\\n                    f.write(\\\"### describe\\\\n```text\\\\n\\\" + matched_describe + \\\"\\\\n```\\\\n\\\")\\n                if matched_logs:\\n                    f.write(\\\"### logs\\\\n```text\\\\n\\\" + matched_logs + \\\"\\\\n```\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join(filter_lines(events, event_kw_list))\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\\\n## Events in Namespace: `{ns}`\\\\n```text\\\\n{events}\\\\n```\\\\n\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-t8pxp/2025/05/08/pod-log-to-github-t8pxp-system-container-impl-418163285"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b0b52bad-4143-40c8-b509-602b3b0542d6\",\"--execution_id\",\"106\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"parameterValues\\\":{\\\"event_keywords\\\":\\\"crash,unhealthy\\\", \\\"log_keywords\\\":\\\"crash,error,fail\\\"}}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b0b52bad-4143-40c8-b509-602b3b0542d6/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"event_keywords\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash,unhealthy\\\", \\\"isOptional\\\":true}, \\\"log_keywords\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash,error,fail\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(\\n    output_report: Output[Dataset],\\n    log_keywords: str = \\\"crash,error,fail\\\",\\n    event_keywords: str = \\\"crash,unhealthy\\\"\\n):\\n    import subprocess\\n    import shutil\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.md\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    def filter_lines(text, keywords):\\n        return [line for line in text.splitlines() if any(k.lower() in line.lower() for k in keywords)]\\n\\n    log_kw_list = [k.strip() for k in log_keywords.split(\\\",\\\") if k.strip()]\\n    event_kw_list = [k.strip() for k in event_keywords.split(\\\",\\\") if k.strip()]\\n\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"# 장애 복기 리포트\\\\n\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join(filter_lines(describe, log_kw_list))\\n            matched_logs = \\\"\\\\n\\\".join(filter_lines(logs, log_kw_list))\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\\\n\\\")\\n                if matched_describe:\\n                    f.write(\\\"### describe\\\\n```text\\\\n\\\" + matched_describe + \\\"\\\\n```\\\\n\\\")\\n                if matched_logs:\\n                    f.write(\\\"### logs\\\\n```text\\\\n\\\" + matched_logs + \\\"\\\\n```\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join(filter_lines(events, event_kw_list))\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\\\n## Events in Namespace: `{ns}`\\\\n```text\\\\n{events}\\\\n```\\\\n\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b0b52bad-4143-40c8-b509-602b3b0542d6\",\"--execution_id\",\"106\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"parameterValues\\\":{\\\"event_keywords\\\":\\\"crash,unhealthy\\\", \\\"log_keywords\\\":\\\"crash,error,fail\\\"}}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b0b52bad-4143-40c8-b509-602b3b0542d6/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"event_keywords\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash,unhealthy\\\", \\\"isOptional\\\":true}, \\\"log_keywords\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash,error,fail\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(\\n    output_report: Output[Dataset],\\n    log_keywords: str = \\\"crash,error,fail\\\",\\n    event_keywords: str = \\\"crash,unhealthy\\\"\\n):\\n    import subprocess\\n    import shutil\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.md\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    def filter_lines(text, keywords):\\n        return [line for line in text.splitlines() if any(k.lower() in line.lower() for k in keywords)]\\n\\n    log_kw_list = [k.strip() for k in log_keywords.split(\\\",\\\") if k.strip()]\\n    event_kw_list = [k.strip() for k in event_keywords.split(\\\",\\\") if k.strip()]\\n\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"# 장애 복기 리포트\\\\n\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join(filter_lines(describe, log_kw_list))\\n            matched_logs = \\\"\\\\n\\\".join(filter_lines(logs, log_kw_list))\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\\\n\\\")\\n                if matched_describe:\\n                    f.write(\\\"### describe\\\\n```text\\\\n\\\" + matched_describe + \\\"\\\\n```\\\\n\\\")\\n                if matched_logs:\\n                    f.write(\\\"### logs\\\\n```text\\\\n\\\" + matched_logs + \\\"\\\\n```\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join(filter_lines(events, event_kw_list))\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\\\n## Events in Namespace: `{ns}`\\\\n```text\\\\n{events}\\\\n```\\\\n\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-t8pxp/2025/05/08/pod-log-to-github-t8pxp-system-container-impl-418163285"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b0b52bad-4143-40c8-b509-602b3b0542d6\",\"--execution_id\",\"106\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"parameterValues\\\":{\\\"event_keywords\\\":\\\"crash,unhealthy\\\", \\\"log_keywords\\\":\\\"crash,error,fail\\\"}}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b0b52bad-4143-40c8-b509-602b3b0542d6/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"event_keywords\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash,unhealthy\\\", \\\"isOptional\\\":true}, \\\"log_keywords\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash,error,fail\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(\\n    output_report: Output[Dataset],\\n    log_keywords: str = \\\"crash,error,fail\\\",\\n    event_keywords: str = \\\"crash,unhealthy\\\"\\n):\\n    import subprocess\\n    import shutil\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.md\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    def filter_lines(text, keywords):\\n        return [line for line in text.splitlines() if any(k.lower() in line.lower() for k in keywords)]\\n\\n    log_kw_list = [k.strip() for k in log_keywords.split(\\\",\\\") if k.strip()]\\n    event_kw_list = [k.strip() for k in event_keywords.split(\\\",\\\") if k.strip()]\\n\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"# 장애 복기 리포트\\\\n\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join(filter_lines(describe, log_kw_list))\\n            matched_logs = \\\"\\\\n\\\".join(filter_lines(logs, log_kw_list))\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\\\n\\\")\\n                if matched_describe:\\n                    f.write(\\\"### describe\\\\n```text\\\\n\\\" + matched_describe + \\\"\\\\n```\\\\n\\\")\\n                if matched_logs:\\n                    f.write(\\\"### logs\\\\n```text\\\\n\\\" + matched_logs + \\\"\\\\n```\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join(filter_lines(events, event_kw_list))\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\\\n## Events in Namespace: `{ns}`\\\\n```text\\\\n{events}\\\\n```\\\\n\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
                  except ValueError:
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b0b52bad-4143-40c8-b509-602b3b0542d6\",\"--execution_id\",\"106\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"parameterValues\\\":{\\\"event_keywords\\\":\\\"crash,unhealthy\\\", \\\"log_keywords\\\":\\\"crash,error,fail\\\"}}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b0b52bad-4143-40c8-b509-602b3b0542d6/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"event_keywords\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash,unhealthy\\\", \\\"isOptional\\\":true}, \\\"log_keywords\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash,error,fail\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(\\n    output_report: Output[Dataset],\\n    log_keywords: str = \\\"crash,error,fail\\\",\\n    event_keywords: str = \\\"crash,unhealthy\\\"\\n):\\n    import subprocess\\n    import shutil\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.md\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    def filter_lines(text, keywords):\\n        return [line for line in text.splitlines() if any(k.lower() in line.lower() for k in keywords)]\\n\\n    log_kw_list = [k.strip() for k in log_keywords.split(\\\",\\\") if k.strip()]\\n    event_kw_list = [k.strip() for k in event_keywords.split(\\\",\\\") if k.strip()]\\n\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"# 장애 복기 리포트\\\\n\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join(filter_lines(describe, log_kw_list))\\n            matched_logs = \\\"\\\\n\\\".join(filter_lines(logs, log_kw_list))\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\\\n\\\")\\n                if matched_describe:\\n                    f.write(\\\"### describe\\\\n```text\\\\n\\\" + matched_describe + \\\"\\\\n```\\\\n\\\")\\n                if matched_logs:\\n                    f.write(\\\"### logs\\\\n```text\\\\n\\\" + matched_logs + \\\"\\\\n```\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join(filter_lines(events, event_kw_list))\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\\\n## Events in Namespace: `{ns}`\\\\n```text\\\\n{events}\\\\n```\\\\n\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-t8pxp/2025/05/08/pod-log-to-github-t8pxp-system-container-impl-418163285"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b0b52bad-4143-40c8-b509-602b3b0542d6\",\"--execution_id\",\"106\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"parameterValues\\\":{\\\"event_keywords\\\":\\\"crash,unhealthy\\\", \\\"log_keywords\\\":\\\"crash,error,fail\\\"}}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b0b52bad-4143-40c8-b509-602b3b0542d6/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"event_keywords\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash,unhealthy\\\", \\\"isOptional\\\":true}, \\\"log_keywords\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash,error,fail\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(\\n    output_report: Output[Dataset],\\n    log_keywords: str = \\\"crash,error,fail\\\",\\n    event_keywords: str = \\\"crash,unhealthy\\\"\\n):\\n    import subprocess\\n    import shutil\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.md\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    def filter_lines(text, keywords):\\n        return [line for line in text.splitlines() if any(k.lower() in line.lower() for k in keywords)]\\n\\n    log_kw_list = [k.strip() for k in log_keywords.split(\\\",\\\") if k.strip()]\\n    event_kw_list = [k.strip() for k in event_keywords.split(\\\",\\\") if k.strip()]\\n\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"# 장애 복기 리포트\\\\n\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join(filter_lines(describe, log_kw_list))\\n            matched_logs = \\\"\\\\n\\\".join(filter_lines(logs, log_kw_list))\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"## Pod: `{pod_name}` (Namespace: `{namespace}`)\\\\n\\\")\\n                if matched_describe:\\n                    f.write(\\\"### describe\\\\n```text\\\\n\\\" + matched_describe + \\\"\\\\n```\\\\n\\\")\\n                if matched_logs:\\n                    f.write(\\\"### logs\\\\n```text\\\\n\\\" + matched_logs + \\\"\\\\n```\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join(filter_lines(events, event_kw_list))\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\\\n## Events in Namespace: `{ns}`\\\\n```text\\\\n{events}\\\\n```\\\\n\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}

Logs (filtered)

        "defaultValue": "crash,error,fail",
            "log_keywords": "crash,error,fail"
time="2025-05-08T04:30:17.985Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T04:30:18.929Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-t8pxp-system-dag-driver-527513263 (Namespace: kubeflow-user-example-com)

Logs (filtered)

        "defaultValue": "crash,error,fail",
    "log_keywords": "crash,error,fail",
I0508 04:28:50.991652      13 driver.go:185] Created execution: id:105  name:"run/b0b52bad-4143-40c8-b509-602b3b0542d6"  type_id:13  type:"system.DAGExecution"  last_known_state:RUNNING  custom_properties:{key:"display_name"  value:{string_value:""}}  custom_properties:{key:"inputs"  value:{struct_value:{fields:{key:"branch"  value:{string_value:"main"}}  fields:{key:"event_keywords"  value:{string_value:"crash,unhealthy"}}  fields:{key:"github_token"  value:{string_value:"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"}}  fields:{key:"log_keywords"  value:{string_value:"crash,error,fail"}}  fields:{key:"repo_url"  value:{string_value:"https://github.com/Rosa1026/Kubeflow.git"}}}}}  custom_properties:{key:"task_name"  value:{string_value:""}}  create_time_since_epoch:1746678530959  last_update_time_since_epoch:1746678530959
time="2025-05-08T04:28:51.626Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-tvwg6-system-container-driver-1487588749 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"134"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","f117d550-f39d-4eff-915b-67528047dd94","--dag_execution_id","134","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-tvwg6/2025/05/08/pod-log-to-github-tvwg6-system-container-driver-1487588749"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"134"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","f117d550-f39d-4eff-915b-67528047dd94","--dag_execution_id","134","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-tvwg6/2025/05/08/pod-log-to-github-tvwg6-system-container-driver-1487588749"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","collect_filtered_logs"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \"crash\"):\n    import shutil\n    import json\n    import subprocess\n    import os\n\n    tmp_report_path = \"/tmp/report.html\"\n    seen_namespaces = set()\n    namespace_events = {}\n\n    # kubectl 설치 방법 변경\n    subprocess.run([\"apt-get\", \"update\", \"-y\"], check=True)\n    subprocess.run([ \n        \"curl\", \"-LO\", \"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\"\n    ], check=True)\n    subprocess.run([\"chmod\", \"+x\", \"kubectl\"], check=True)\n    subprocess.run([\"mv\", \"kubectl\", \"/usr/local/bin/\"], check=True)\n\n    # Get all pods in all namespaces\n    pods_output = subprocess.getoutput(\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\")\n    pod_lines = pods_output.strip().split(\"\\n\")\n\n    with open(tmp_report_path, \"w\") as f:\n        f.write(f\"\u003chtml\u003e\u003cbody\u003e\u003ch1\u003e장애 복기 리포트 (keyword: `{keyword}`)\u003c/h1\u003e\\n\")\n\n        for line in pod_lines:\n            if not line.strip():\n                continue\n            try:\n                namespace, pod_name = line.split()\n            except ValueError:\n                continue\n\n            describe = subprocess.getoutput(f\"kubectl describe pod {pod_name} -n {namespace}\")\n            logs = subprocess.getoutput(f\"kubectl logs {pod_name} -n {namespace}\")\n\n            matched_describe = \"\\n\".join([l for l in describe.splitlines() if keyword in l])\n            matched_logs = \"\\n\".join([l for l in logs.splitlines() if keyword in l])\n\n            if matched_describe or matched_logs:\n                f.write(f\"\u003ch2\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\u003c/h2\u003e\\n\")\n                if matched_describe:\n                    f.write(f\"\u003ch3\u003eDescribe\u003c/h3\u003e\u003cpre\u003e{matched_describe}\u003c/pre\u003e\\n\")\n                if matched_logs:\n                    f.write(f\"\u003ch3\u003eLogs\u003c/h3\u003e\u003cpre\u003e{matched_logs}\u003c/pre\u003e\\n\")\n\n            if namespace not in seen_namespaces:\n                events = subprocess.getoutput(f\"kubectl get events -n {namespace}\")\n                matched_events = \"\\n\".join([l for l in events.splitlines() if keyword in l])\n                if matched_events:\n                    namespace_events[namespace] = matched_events\n                seen_namespaces.add(namespace)\n\n        # Write filtered events grouped by namespace\n        for ns, events in namespace_events.items():\n            f.write(f\"\u003ch3\u003eEvents in Namespace: `{ns}`\u003c/h3\u003e\u003cpre\u003e{events}\u003c/pre\u003e\\n\")\n\n        f.write(\"\u003c/body\u003e\u003c/html\u003e\")\n\n    subprocess.run([\"cp\", tmp_report_path, output_report.path], check=True)\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"134"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","f117d550-f39d-4eff-915b-67528047dd94","--dag_execution_id","134","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-tvwg6/2025/05/08/pod-log-to-github-tvwg6-system-container-driver-1487588749"}}}

Logs (filtered)

time="2025-05-08T05:58:02.020Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-tvwg6-system-container-driver-2710516570 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"134"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","f117d550-f39d-4eff-915b-67528047dd94","--dag_execution_id","134","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-tvwg6/2025/05/08/pod-log-to-github-tvwg6-system-container-driver-2710516570"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"134"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","f117d550-f39d-4eff-915b-67528047dd94","--dag_execution_id","134","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-tvwg6/2025/05/08/pod-log-to-github-tvwg6-system-container-driver-2710516570"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str\n):\n\n    import subprocess\n    import shutil\n    import os\n    from datetime import datetime\n\n    os.environ[\"GIT_ASKPASS\"] = \"echo\"\n\n    report_path = report_file.path\n    print(\"Checking if report exists at:\", report_path)\n    if not os.path.exists(report_path):\n        raise FileNotFoundError(f\"{report_path} does not exist\")\n\n    # HTML 파일 이름 만들기\n    timestamp = datetime.now().strftime(\"report-%Y%m%d-%H%M%S.html\")\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n    report_target_path = os.path.join(repo_dir, timestamp)\n\n    # Git clone\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    shutil.copyfile(report_path, report_target_path)\n\n    # Git config\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    # Git commit \u0026 push\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", f\"Add {timestamp}\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(f\"Report committed and pushed to GitHub as {timestamp}.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"134"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","f117d550-f39d-4eff-915b-67528047dd94","--dag_execution_id","134","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-tvwg6/2025/05/08/pod-log-to-github-tvwg6-system-container-driver-2710516570"}}}

Logs (filtered)

time="2025-05-08T05:59:40.216Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-tvwg6-system-container-impl-3280166619 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"f117d550-f39d-4eff-915b-67528047dd94\",\"--execution_id\",\"135\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/f117d550-f39d-4eff-915b-67528047dd94/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-tvwg6/2025/05/08/pod-log-to-github-tvwg6-system-container-impl-3280166619"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"f117d550-f39d-4eff-915b-67528047dd94\",\"--execution_id\",\"135\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/f117d550-f39d-4eff-915b-67528047dd94/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"f117d550-f39d-4eff-915b-67528047dd94\",\"--execution_id\",\"135\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/f117d550-f39d-4eff-915b-67528047dd94/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-tvwg6/2025/05/08/pod-log-to-github-tvwg6-system-container-impl-3280166619"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"f117d550-f39d-4eff-915b-67528047dd94\",\"--execution_id\",\"135\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/f117d550-f39d-4eff-915b-67528047dd94/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"f117d550-f39d-4eff-915b-67528047dd94\",\"--execution_id\",\"135\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/f117d550-f39d-4eff-915b-67528047dd94/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-tvwg6/2025/05/08/pod-log-to-github-tvwg6-system-container-impl-3280166619"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"f117d550-f39d-4eff-915b-67528047dd94\",\"--execution_id\",\"135\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/f117d550-f39d-4eff-915b-67528047dd94/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
                  except ValueError:
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"f117d550-f39d-4eff-915b-67528047dd94\",\"--execution_id\",\"135\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/f117d550-f39d-4eff-915b-67528047dd94/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-tvwg6/2025/05/08/pod-log-to-github-tvwg6-system-container-impl-3280166619"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"f117d550-f39d-4eff-915b-67528047dd94\",\"--execution_id\",\"135\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/f117d550-f39d-4eff-915b-67528047dd94/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}

Logs (filtered)

time="2025-05-08T05:59:26.254Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T05:59:27.202Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-tvwg6-system-dag-driver-2279793957 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T05:57:51.926Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-vc5rx-system-container-driver-854133989 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T07:56:33.014Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-vc5rx-system-container-impl-1672389891 (Namespace: kubeflow-user-example-com)

Describe (filtered)

Status:       Failed
      Reason:       Error

Logs (filtered)

Traceback (most recent call last):
F0507 07:56:52.673249      24 main.go:49] failed to execute component: exit status 1
time="2025-05-07T07:56:53.125Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-07T07:56:54.065Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-vc5rx-system-dag-driver-3256936155 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T07:56:22.915Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-vkpgx-system-container-driver-3905965553 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T00:31:40.837Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-vkpgx-system-container-impl-1923254719 (Namespace: kubeflow-user-example-com)

Describe (filtered)

Status:       Failed
      Reason:       Error

Logs (filtered)

Traceback (most recent call last):
F0508 00:32:06.210781      23 main.go:49] failed to execute component: exit status 1
time="2025-05-08T00:32:07.062Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T00:32:07.999Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-vkpgx-system-dag-driver-3498199805 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T00:31:30.737Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-vpkhs-system-container-driver-1605493575 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T07:36:07.024Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-vpkhs-system-container-impl-2059144489 (Namespace: kubeflow-user-example-com)

Describe (filtered)

Status:       Failed
      Reason:       Error

Logs (filtered)

F0507 07:36:17.366029      24 main.go:49] failed to execute component: unable to create directory "/minio/mlpipeline/v2/artifacts/pod-log-to-github/14a4bb55-0ac3-4dda-bd36-d4293e3723ae/collect-pod-logs" for output artifact "output_report": mkdir /minio: permission denied
time="2025-05-07T07:36:18.157Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-07T07:36:19.097Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-vpkhs-system-dag-driver-3991735641 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T07:35:57.932Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-vrd87-system-container-driver-3290906743 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"64"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","667977d4-5b85-41f3-aae9-2592dc079214","--dag_execution_id","64","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-vrd87/2025/05/07/pod-log-to-github-vrd87-system-container-driver-3290906743"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"64"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","667977d4-5b85-41f3-aae9-2592dc079214","--dag_execution_id","64","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-vrd87/2025/05/07/pod-log-to-github-vrd87-system-container-driver-3290906743"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str = \"main\"\n):\n    import os\n    import subprocess\n\n\n    report_path = report_file.path\n    print(\"Checking if report exists at:\", report_path)\n    if not os.path.exists(report_path):\n        raise FileNotFoundError(f\"{report_path} does not exist\")\n\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    subprocess.run([\"cp\", report_path, os.path.join(repo_dir, \"report.md\")], check=True)\n\n    # Git user config\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", \"Add report\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(\"Report committed and pushed to GitHub.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"64"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","667977d4-5b85-41f3-aae9-2592dc079214","--dag_execution_id","64","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-vrd87/2025/05/07/pod-log-to-github-vrd87-system-container-driver-3290906743"}}}

Logs (filtered)

time="2025-05-07T08:14:15.911Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-07T08:14:15.911Z" level=error msg="cannot save parameter /tmp/outputs/pod-spec-patch" argo=true error="open /tmp/outputs/pod-spec-patch: no such file or directory"

Pod: pod-log-to-github-vrd87-system-container-driver-480125082 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T08:14:05.720Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-07T08:14:05.720Z" level=error msg="cannot save parameter /tmp/outputs/pod-spec-patch" argo=true error="open /tmp/outputs/pod-spec-patch: no such file or directory"

Pod: pod-log-to-github-vrd87-system-dag-driver-3105100070 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T08:13:55.619Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-x58xk-system-container-driver-2167843561 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T07:53:31.719Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-07T07:53:31.719Z" level=error msg="cannot save parameter /tmp/outputs/pod-spec-patch" argo=true error="open /tmp/outputs/pod-spec-patch: no such file or directory"

Pod: pod-log-to-github-x58xk-system-container-driver-2304913460 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"53"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","1b168959-a4ff-4e47-a7e6-5266d70c6278","--dag_execution_id","53","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-x58xk/2025/05/07/pod-log-to-github-x58xk-system-container-driver-2304913460"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"53"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","1b168959-a4ff-4e47-a7e6-5266d70c6278","--dag_execution_id","53","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-x58xk/2025/05/07/pod-log-to-github-x58xk-system-container-driver-2304913460"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str = \"main\"\n):\n    import os\n    import subprocess\n\n\n    report_path = report_file.path\n    print(\"Checking if report exists at:\", report_path)\n    if not os.path.exists(report_path):\n        raise FileNotFoundError(f\"{report_path} does not exist\")\n\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    subprocess.run([\"cp\", report_path, os.path.join(repo_dir, \"report.md\")], check=True)\n\n    # Git user config\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", \"Add report\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(\"Report committed and pushed to GitHub.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"53"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","1b168959-a4ff-4e47-a7e6-5266d70c6278","--dag_execution_id","53","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-x58xk/2025/05/07/pod-log-to-github-x58xk-system-container-driver-2304913460"}}}

Logs (filtered)

time="2025-05-07T07:53:41.830Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-x58xk-system-container-impl-1070925274 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1b168959-a4ff-4e47-a7e6-5266d70c6278\",\"--execution_id\",\"55\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"2\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ea0f30be-c0ab-40be-a31d-b6768793aa19/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-x58xk/2025/05/07/pod-log-to-github-x58xk-system-container-impl-1070925274"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1b168959-a4ff-4e47-a7e6-5266d70c6278\",\"--execution_id\",\"55\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"2\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ea0f30be-c0ab-40be-a31d-b6768793aa19/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1b168959-a4ff-4e47-a7e6-5266d70c6278\",\"--execution_id\",\"55\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"2\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ea0f30be-c0ab-40be-a31d-b6768793aa19/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-x58xk/2025/05/07/pod-log-to-github-x58xk-system-container-impl-1070925274"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1b168959-a4ff-4e47-a7e6-5266d70c6278\",\"--execution_id\",\"55\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"2\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ea0f30be-c0ab-40be-a31d-b6768793aa19/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1b168959-a4ff-4e47-a7e6-5266d70c6278\",\"--execution_id\",\"55\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"2\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ea0f30be-c0ab-40be-a31d-b6768793aa19/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-x58xk/2025/05/07/pod-log-to-github-x58xk-system-container-impl-1070925274"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1b168959-a4ff-4e47-a7e6-5266d70c6278\",\"--execution_id\",\"55\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"2\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ea0f30be-c0ab-40be-a31d-b6768793aa19/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
              raise FileNotFoundError(f"{report_path} does not exist")
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1b168959-a4ff-4e47-a7e6-5266d70c6278\",\"--execution_id\",\"55\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"2\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ea0f30be-c0ab-40be-a31d-b6768793aa19/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-x58xk/2025/05/07/pod-log-to-github-x58xk-system-container-impl-1070925274"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1b168959-a4ff-4e47-a7e6-5266d70c6278\",\"--execution_id\",\"55\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"2\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/ea0f30be-c0ab-40be-a31d-b6768793aa19/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}

Logs (filtered)

time="2025-05-07T07:54:02.017Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-07T07:54:02.955Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-x58xk-system-dag-driver-2021631647 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T07:53:21.625Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-x8pgq-system-container-driver-2228026073 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T07:59:03.046Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-x8pgq-system-container-driver-590905124 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"58"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","81b66e70-195d-40d9-ab4c-8a8eeabedd96","--dag_execution_id","58","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-x8pgq/2025/05/07/pod-log-to-github-x8pgq-system-container-driver-590905124"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"58"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","81b66e70-195d-40d9-ab4c-8a8eeabedd96","--dag_execution_id","58","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-x8pgq/2025/05/07/pod-log-to-github-x8pgq-system-container-driver-590905124"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str = \"main\"\n):\n    import os\n    import subprocess\n\n\n    report_path = report_file.path\n    print(\"Checking if report exists at:\", report_path)\n    if not os.path.exists(report_path):\n        raise FileNotFoundError(f\"{report_path} does not exist\")\n\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    subprocess.run([\"cp\", report_path, os.path.join(repo_dir, \"report.md\")], check=True)\n\n    # Git user config\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", \"Add report\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(\"Report committed and pushed to GitHub.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"58"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","81b66e70-195d-40d9-ab4c-8a8eeabedd96","--dag_execution_id","58","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-pod-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-pod-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-x8pgq/2025/05/07/pod-log-to-github-x8pgq-system-container-driver-590905124"}}}

Logs (filtered)

time="2025-05-07T07:59:42.425Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-x8pgq-system-container-impl-3555163882 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"81b66e70-195d-40d9-ab4c-8a8eeabedd96\",\"--execution_id\",\"60\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"3\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/81b66e70-195d-40d9-ab4c-8a8eeabedd96/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-x8pgq/2025/05/07/pod-log-to-github-x8pgq-system-container-impl-3555163882"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"81b66e70-195d-40d9-ab4c-8a8eeabedd96\",\"--execution_id\",\"60\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"3\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/81b66e70-195d-40d9-ab4c-8a8eeabedd96/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"81b66e70-195d-40d9-ab4c-8a8eeabedd96\",\"--execution_id\",\"60\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"3\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/81b66e70-195d-40d9-ab4c-8a8eeabedd96/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-x8pgq/2025/05/07/pod-log-to-github-x8pgq-system-container-impl-3555163882"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"81b66e70-195d-40d9-ab4c-8a8eeabedd96\",\"--execution_id\",\"60\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"3\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/81b66e70-195d-40d9-ab4c-8a8eeabedd96/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"81b66e70-195d-40d9-ab4c-8a8eeabedd96\",\"--execution_id\",\"60\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"3\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/81b66e70-195d-40d9-ab4c-8a8eeabedd96/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-x8pgq/2025/05/07/pod-log-to-github-x8pgq-system-container-impl-3555163882"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"81b66e70-195d-40d9-ab4c-8a8eeabedd96\",\"--execution_id\",\"60\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"3\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/81b66e70-195d-40d9-ab4c-8a8eeabedd96/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
              raise FileNotFoundError(f"{report_path} does not exist")
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"81b66e70-195d-40d9-ab4c-8a8eeabedd96\",\"--execution_id\",\"60\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"3\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/81b66e70-195d-40d9-ab4c-8a8eeabedd96/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-x8pgq/2025/05/07/pod-log-to-github-x8pgq-system-container-impl-3555163882"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"81b66e70-195d-40d9-ab4c-8a8eeabedd96\",\"--execution_id\",\"60\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"3\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/81b66e70-195d-40d9-ab4c-8a8eeabedd96/collect-pod-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n    import os\\n    import subprocess\\n\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    subprocess.run([\\\"cp\\\", report_path, os.path.join(repo_dir, \\\"report.md\\\")], check=True)\\n\\n    # Git user config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", \\\"Add report\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(\\\"Report committed and pushed to GitHub.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}

Logs (filtered)

time="2025-05-07T08:00:00.582Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-07T08:00:01.518Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-x8pgq-system-container-impl-536717815 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T07:59:28.322Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-07T07:59:29.263Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-x8pgq-system-dag-driver-3742823439 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T07:58:54.029Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-xk99f-system-container-driver-2553311145 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"143"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","39b9d1ea-c847-4bba-ac72-2975413170e4","--dag_execution_id","143","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-xk99f/2025/05/08/pod-log-to-github-xk99f-system-container-driver-2553311145"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"143"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","39b9d1ea-c847-4bba-ac72-2975413170e4","--dag_execution_id","143","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-xk99f/2025/05/08/pod-log-to-github-xk99f-system-container-driver-2553311145"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str\n):\n\n    import subprocess\n    import shutil\n    import os\n    from datetime import datetime\n\n    os.environ[\"GIT_ASKPASS\"] = \"echo\"\n\n    report_path = report_file.path\n    print(\"Checking if report exists at:\", report_path)\n    if not os.path.exists(report_path):\n        raise FileNotFoundError(f\"{report_path} does not exist\")\n\n    # HTML 파일 이름 만들기\n    timestamp = datetime.now().strftime(\"report-%Y%m%d-%H%M%S.html\")\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n    report_target_path = os.path.join(repo_dir, timestamp)\n\n    # Git clone\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    shutil.copyfile(report_path, report_target_path)\n\n    # Git config\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    # Git commit \u0026 push\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", f\"Add {timestamp}\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(f\"Report committed and pushed to GitHub as {timestamp}.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"143"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","39b9d1ea-c847-4bba-ac72-2975413170e4","--dag_execution_id","143","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-xk99f/2025/05/08/pod-log-to-github-xk99f-system-container-driver-2553311145"}}}

Logs (filtered)

time="2025-05-08T06:19:42.045Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-xk99f-system-container-driver-537797304 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n    report_data = []\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # 모든 Pod 가져오기\\n    pods_output = subprocess.getoutput(\\n        \\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\"\\n    )\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    for line in pod_lines:\\n        if not line.strip():\\n            continue\\n        try:\\n            namespace, pod_name = line.split()\\n        except ValueError:\\n            continue\\n\\n        describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n        logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n        matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n        matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n        if matched_describe or matched_logs:\\n            report_data.append({\\n                \\\"namespace\\\": namespace,\\n                \\\"pod\\\": pod_name,\\n                \\\"describe\\\": matched_describe,\\n                \\\"logs\\\": matched_logs\\n            })\\n\\n        if namespace not in seen_namespaces:\\n            events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n            matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n            if matched_events:\\n                namespace_events[namespace] = matched_events\\n            seen_namespaces.add(namespace)\\n\\n    # HTML 작성\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(\\\"\\\"\\\"\\n\\u003c!DOCTYPE html\\u003e\\n\\u003chtml lang=\\\"en\\\"\\u003e\\n\\u003chead\\u003e\\n    \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n    \\u003ctitle\\u003eKubeflow Log Report\\u003c/title\\u003e\\n    \\u003cstyle\\u003e\\n        body {\\n            font-family: 'Segoe UI', sans-serif;\\n            background-color: #f9f9f9;\\n            color: #333;\\n            padding: 2em;\\n            line-height: 1.6;\\n        }\\n        h1 {\\n            color: #004080;\\n            text-align: center;\\n        }\\n        .namespace {\\n            margin-bottom: 2em;\\n            border: 1px solid #ccc;\\n            padding: 1em;\\n            background-color: #fff;\\n            box-shadow: 0 2px 5px rgba(0,0,0,0.1);\\n            border-radius: 8px;\\n        }\\n        .pod-name {\\n            font-weight: bold;\\n            margin-top: 1em;\\n            font-size: 1.2em;\\n            color: #2a5885;\\n        }\\n        .section-title {\\n            margin-top: 1em;\\n            color: #555;\\n            font-size: 1em;\\n            border-bottom: 1px solid #ddd;\\n        }\\n        pre {\\n            background-color: #272822;\\n            color: #f8f8f2;\\n            padding: 1em;\\n            overflow-x: auto;\\n            white-space: pre-wrap;\\n            border-radius: 4px;\\n        }\\n        .events {\\n            background-color: #fffaf0;\\n            border-left: 5px solid #ffa500;\\n            padding: 1em;\\n            margin-top: 1em;\\n        }\\n    \\u003c/style\\u003e\\n\\u003c/head\\u003e\\n\\u003cbody\\u003e\\n\\u003ch1\\u003eKubeflow Pod Diagnostics Report\\u003c/h1\\u003e\\n\\\"\\\"\\\")\\n\\n        for entry in report_data:\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003ePod: {entry[\\\"pod\\\"]} (Namespace: {entry[\\\"namespace\\\"]})\\u003c/div\\u003e\\\\n')\\n            if entry.get(\\\"describe\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eDescribe\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"describe\\\"]}\\u003c/pre\\u003e\\\\n')\\n            if entry.get(\\\"logs\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eLogs\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"logs\\\"]}\\u003c/pre\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003eEvents in Namespace: {ns}\\u003c/div\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"events\\\"\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/div\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"143"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","39b9d1ea-c847-4bba-ac72-2975413170e4","--dag_execution_id","143","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n    report_data = []\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # 모든 Pod 가져오기\\n    pods_output = subprocess.getoutput(\\n        \\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\"\\n    )\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    for line in pod_lines:\\n        if not line.strip():\\n            continue\\n        try:\\n            namespace, pod_name = line.split()\\n        except ValueError:\\n            continue\\n\\n        describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n        logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n        matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n        matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n        if matched_describe or matched_logs:\\n            report_data.append({\\n                \\\"namespace\\\": namespace,\\n                \\\"pod\\\": pod_name,\\n                \\\"describe\\\": matched_describe,\\n                \\\"logs\\\": matched_logs\\n            })\\n\\n        if namespace not in seen_namespaces:\\n            events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n            matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n            if matched_events:\\n                namespace_events[namespace] = matched_events\\n            seen_namespaces.add(namespace)\\n\\n    # HTML 작성\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(\\\"\\\"\\\"\\n\\u003c!DOCTYPE html\\u003e\\n\\u003chtml lang=\\\"en\\\"\\u003e\\n\\u003chead\\u003e\\n    \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n    \\u003ctitle\\u003eKubeflow Log Report\\u003c/title\\u003e\\n    \\u003cstyle\\u003e\\n        body {\\n            font-family: 'Segoe UI', sans-serif;\\n            background-color: #f9f9f9;\\n            color: #333;\\n            padding: 2em;\\n            line-height: 1.6;\\n        }\\n        h1 {\\n            color: #004080;\\n            text-align: center;\\n        }\\n        .namespace {\\n            margin-bottom: 2em;\\n            border: 1px solid #ccc;\\n            padding: 1em;\\n            background-color: #fff;\\n            box-shadow: 0 2px 5px rgba(0,0,0,0.1);\\n            border-radius: 8px;\\n        }\\n        .pod-name {\\n            font-weight: bold;\\n            margin-top: 1em;\\n            font-size: 1.2em;\\n            color: #2a5885;\\n        }\\n        .section-title {\\n            margin-top: 1em;\\n            color: #555;\\n            font-size: 1em;\\n            border-bottom: 1px solid #ddd;\\n        }\\n        pre {\\n            background-color: #272822;\\n            color: #f8f8f2;\\n            padding: 1em;\\n            overflow-x: auto;\\n            white-space: pre-wrap;\\n            border-radius: 4px;\\n        }\\n        .events {\\n            background-color: #fffaf0;\\n            border-left: 5px solid #ffa500;\\n            padding: 1em;\\n            margin-top: 1em;\\n        }\\n    \\u003c/style\\u003e\\n\\u003c/head\\u003e\\n\\u003cbody\\u003e\\n\\u003ch1\\u003eKubeflow Pod Diagnostics Report\\u003c/h1\\u003e\\n\\\"\\\"\\\")\\n\\n        for entry in report_data:\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003ePod: {entry[\\\"pod\\\"]} (Namespace: {entry[\\\"namespace\\\"]})\\u003c/div\\u003e\\\\n')\\n            if entry.get(\\\"describe\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eDescribe\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"describe\\\"]}\\u003c/pre\\u003e\\\\n')\\n            if entry.get(\\\"logs\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eLogs\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"logs\\\"]}\\u003c/pre\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003eEvents in Namespace: {ns}\\u003c/div\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"events\\\"\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/div\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-xk99f/2025/05/08/pod-log-to-github-xk99f-system-container-driver-537797304"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n    report_data = []\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # 모든 Pod 가져오기\\n    pods_output = subprocess.getoutput(\\n        \\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\"\\n    )\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    for line in pod_lines:\\n        if not line.strip():\\n            continue\\n        try:\\n            namespace, pod_name = line.split()\\n        except ValueError:\\n            continue\\n\\n        describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n        logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n        matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n        matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n        if matched_describe or matched_logs:\\n            report_data.append({\\n                \\\"namespace\\\": namespace,\\n                \\\"pod\\\": pod_name,\\n                \\\"describe\\\": matched_describe,\\n                \\\"logs\\\": matched_logs\\n            })\\n\\n        if namespace not in seen_namespaces:\\n            events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n            matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n            if matched_events:\\n                namespace_events[namespace] = matched_events\\n            seen_namespaces.add(namespace)\\n\\n    # HTML 작성\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(\\\"\\\"\\\"\\n\\u003c!DOCTYPE html\\u003e\\n\\u003chtml lang=\\\"en\\\"\\u003e\\n\\u003chead\\u003e\\n    \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n    \\u003ctitle\\u003eKubeflow Log Report\\u003c/title\\u003e\\n    \\u003cstyle\\u003e\\n        body {\\n            font-family: 'Segoe UI', sans-serif;\\n            background-color: #f9f9f9;\\n            color: #333;\\n            padding: 2em;\\n            line-height: 1.6;\\n        }\\n        h1 {\\n            color: #004080;\\n            text-align: center;\\n        }\\n        .namespace {\\n            margin-bottom: 2em;\\n            border: 1px solid #ccc;\\n            padding: 1em;\\n            background-color: #fff;\\n            box-shadow: 0 2px 5px rgba(0,0,0,0.1);\\n            border-radius: 8px;\\n        }\\n        .pod-name {\\n            font-weight: bold;\\n            margin-top: 1em;\\n            font-size: 1.2em;\\n            color: #2a5885;\\n        }\\n        .section-title {\\n            margin-top: 1em;\\n            color: #555;\\n            font-size: 1em;\\n            border-bottom: 1px solid #ddd;\\n        }\\n        pre {\\n            background-color: #272822;\\n            color: #f8f8f2;\\n            padding: 1em;\\n            overflow-x: auto;\\n            white-space: pre-wrap;\\n            border-radius: 4px;\\n        }\\n        .events {\\n            background-color: #fffaf0;\\n            border-left: 5px solid #ffa500;\\n            padding: 1em;\\n            margin-top: 1em;\\n        }\\n    \\u003c/style\\u003e\\n\\u003c/head\\u003e\\n\\u003cbody\\u003e\\n\\u003ch1\\u003eKubeflow Pod Diagnostics Report\\u003c/h1\\u003e\\n\\\"\\\"\\\")\\n\\n        for entry in report_data:\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003ePod: {entry[\\\"pod\\\"]} (Namespace: {entry[\\\"namespace\\\"]})\\u003c/div\\u003e\\\\n')\\n            if entry.get(\\\"describe\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eDescribe\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"describe\\\"]}\\u003c/pre\\u003e\\\\n')\\n            if entry.get(\\\"logs\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eLogs\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"logs\\\"]}\\u003c/pre\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003eEvents in Namespace: {ns}\\u003c/div\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"events\\\"\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/div\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"143"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","39b9d1ea-c847-4bba-ac72-2975413170e4","--dag_execution_id","143","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n    report_data = []\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # 모든 Pod 가져오기\\n    pods_output = subprocess.getoutput(\\n        \\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\"\\n    )\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    for line in pod_lines:\\n        if not line.strip():\\n            continue\\n        try:\\n            namespace, pod_name = line.split()\\n        except ValueError:\\n            continue\\n\\n        describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n        logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n        matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n        matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n        if matched_describe or matched_logs:\\n            report_data.append({\\n                \\\"namespace\\\": namespace,\\n                \\\"pod\\\": pod_name,\\n                \\\"describe\\\": matched_describe,\\n                \\\"logs\\\": matched_logs\\n            })\\n\\n        if namespace not in seen_namespaces:\\n            events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n            matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n            if matched_events:\\n                namespace_events[namespace] = matched_events\\n            seen_namespaces.add(namespace)\\n\\n    # HTML 작성\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(\\\"\\\"\\\"\\n\\u003c!DOCTYPE html\\u003e\\n\\u003chtml lang=\\\"en\\\"\\u003e\\n\\u003chead\\u003e\\n    \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n    \\u003ctitle\\u003eKubeflow Log Report\\u003c/title\\u003e\\n    \\u003cstyle\\u003e\\n        body {\\n            font-family: 'Segoe UI', sans-serif;\\n            background-color: #f9f9f9;\\n            color: #333;\\n            padding: 2em;\\n            line-height: 1.6;\\n        }\\n        h1 {\\n            color: #004080;\\n            text-align: center;\\n        }\\n        .namespace {\\n            margin-bottom: 2em;\\n            border: 1px solid #ccc;\\n            padding: 1em;\\n            background-color: #fff;\\n            box-shadow: 0 2px 5px rgba(0,0,0,0.1);\\n            border-radius: 8px;\\n        }\\n        .pod-name {\\n            font-weight: bold;\\n            margin-top: 1em;\\n            font-size: 1.2em;\\n            color: #2a5885;\\n        }\\n        .section-title {\\n            margin-top: 1em;\\n            color: #555;\\n            font-size: 1em;\\n            border-bottom: 1px solid #ddd;\\n        }\\n        pre {\\n            background-color: #272822;\\n            color: #f8f8f2;\\n            padding: 1em;\\n            overflow-x: auto;\\n            white-space: pre-wrap;\\n            border-radius: 4px;\\n        }\\n        .events {\\n            background-color: #fffaf0;\\n            border-left: 5px solid #ffa500;\\n            padding: 1em;\\n            margin-top: 1em;\\n        }\\n    \\u003c/style\\u003e\\n\\u003c/head\\u003e\\n\\u003cbody\\u003e\\n\\u003ch1\\u003eKubeflow Pod Diagnostics Report\\u003c/h1\\u003e\\n\\\"\\\"\\\")\\n\\n        for entry in report_data:\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003ePod: {entry[\\\"pod\\\"]} (Namespace: {entry[\\\"namespace\\\"]})\\u003c/div\\u003e\\\\n')\\n            if entry.get(\\\"describe\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eDescribe\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"describe\\\"]}\\u003c/pre\\u003e\\\\n')\\n            if entry.get(\\\"logs\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eLogs\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"logs\\\"]}\\u003c/pre\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003eEvents in Namespace: {ns}\\u003c/div\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"events\\\"\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/div\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-xk99f/2025/05/08/pod-log-to-github-xk99f-system-container-driver-537797304"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","collect_filtered_logs"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \"crash\"):\n    import shutil\n    import subprocess\n    import os\n\n    tmp_report_path = \"/tmp/report.html\"\n    seen_namespaces = set()\n    namespace_events = {}\n    report_data = []\n\n    # kubectl 설치\n    subprocess.run([\"apt-get\", \"update\", \"-y\"], check=True)\n    subprocess.run([\n        \"curl\", \"-LO\", \"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\"\n    ], check=True)\n    subprocess.run([\"chmod\", \"+x\", \"kubectl\"], check=True)\n    subprocess.run([\"mv\", \"kubectl\", \"/usr/local/bin/\"], check=True)\n\n    # 모든 Pod 가져오기\n    pods_output = subprocess.getoutput(\n        \"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\"\n    )\n    pod_lines = pods_output.strip().split(\"\\n\")\n\n    for line in pod_lines:\n        if not line.strip():\n            continue\n        try:\n            namespace, pod_name = line.split()\n        except ValueError:\n            continue\n\n        describe = subprocess.getoutput(f\"kubectl describe pod {pod_name} -n {namespace}\")\n        logs = subprocess.getoutput(f\"kubectl logs {pod_name} -n {namespace}\")\n\n        matched_describe = \"\\n\".join([l for l in describe.splitlines() if keyword in l])\n        matched_logs = \"\\n\".join([l for l in logs.splitlines() if keyword in l])\n\n        if matched_describe or matched_logs:\n            report_data.append({\n                \"namespace\": namespace,\n                \"pod\": pod_name,\n                \"describe\": matched_describe,\n                \"logs\": matched_logs\n            })\n\n        if namespace not in seen_namespaces:\n            events = subprocess.getoutput(f\"kubectl get events -n {namespace}\")\n            matched_events = \"\\n\".join([l for l in events.splitlines() if keyword in l])\n            if matched_events:\n                namespace_events[namespace] = matched_events\n            seen_namespaces.add(namespace)\n\n    # HTML 작성\n    with open(tmp_report_path, \"w\") as f:\n        f.write(\"\"\"\n\u003c!DOCTYPE html\u003e\n\u003chtml lang=\"en\"\u003e\n\u003chead\u003e\n    \u003cmeta charset=\"UTF-8\"\u003e\n    \u003ctitle\u003eKubeflow Log Report\u003c/title\u003e\n    \u003cstyle\u003e\n        body {\n            font-family: 'Segoe UI', sans-serif;\n            background-color: #f9f9f9;\n            color: #333;\n            padding: 2em;\n            line-height: 1.6;\n        }\n        h1 {\n            color: #004080;\n            text-align: center;\n        }\n        .namespace {\n            margin-bottom: 2em;\n            border: 1px solid #ccc;\n            padding: 1em;\n            background-color: #fff;\n            box-shadow: 0 2px 5px rgba(0,0,0,0.1);\n            border-radius: 8px;\n        }\n        .pod-name {\n            font-weight: bold;\n            margin-top: 1em;\n            font-size: 1.2em;\n            color: #2a5885;\n        }\n        .section-title {\n            margin-top: 1em;\n            color: #555;\n            font-size: 1em;\n            border-bottom: 1px solid #ddd;\n        }\n        pre {\n            background-color: #272822;\n            color: #f8f8f2;\n            padding: 1em;\n            overflow-x: auto;\n            white-space: pre-wrap;\n            border-radius: 4px;\n        }\n        .events {\n            background-color: #fffaf0;\n            border-left: 5px solid #ffa500;\n            padding: 1em;\n            margin-top: 1em;\n        }\n    \u003c/style\u003e\n\u003c/head\u003e\n\u003cbody\u003e\n\u003ch1\u003eKubeflow Pod Diagnostics Report\u003c/h1\u003e\n\"\"\")\n\n        for entry in report_data:\n            f.write(f'\u003cdiv class=\"namespace\"\u003e\\n')\n            f.write(f'\u003cdiv class=\"pod-name\"\u003ePod: {entry[\"pod\"]} (Namespace: {entry[\"namespace\"]})\u003c/div\u003e\\n')\n            if entry.get(\"describe\"):\n                f.write(f'\u003cdiv class=\"section-title\"\u003eDescribe\u003c/div\u003e\\n\u003cpre\u003e{entry[\"describe\"]}\u003c/pre\u003e\\n')\n            if entry.get(\"logs\"):\n                f.write(f'\u003cdiv class=\"section-title\"\u003eLogs\u003c/div\u003e\\n\u003cpre\u003e{entry[\"logs\"]}\u003c/pre\u003e\\n')\n            f.write('\u003c/div\u003e\\n')\n\n        for ns, events in namespace_events.items():\n            f.write(f'\u003cdiv class=\"namespace\"\u003e\\n')\n            f.write(f'\u003cdiv class=\"pod-name\"\u003eEvents in Namespace: {ns}\u003c/div\u003e\\n')\n            f.write(f'\u003cdiv class=\"events\"\u003e\u003cpre\u003e{events}\u003c/pre\u003e\u003c/div\u003e\\n')\n            f.write('\u003c/div\u003e\\n')\n\n        f.write(\"\u003c/body\u003e\u003c/html\u003e\")\n\n    subprocess.run([\"cp\", tmp_report_path, output_report.path], check=True)\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n    report_data = []\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # 모든 Pod 가져오기\\n    pods_output = subprocess.getoutput(\\n        \\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\"\\n    )\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    for line in pod_lines:\\n        if not line.strip():\\n            continue\\n        try:\\n            namespace, pod_name = line.split()\\n        except ValueError:\\n            continue\\n\\n        describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n        logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n        matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n        matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n        if matched_describe or matched_logs:\\n            report_data.append({\\n                \\\"namespace\\\": namespace,\\n                \\\"pod\\\": pod_name,\\n                \\\"describe\\\": matched_describe,\\n                \\\"logs\\\": matched_logs\\n            })\\n\\n        if namespace not in seen_namespaces:\\n            events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n            matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n            if matched_events:\\n                namespace_events[namespace] = matched_events\\n            seen_namespaces.add(namespace)\\n\\n    # HTML 작성\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(\\\"\\\"\\\"\\n\\u003c!DOCTYPE html\\u003e\\n\\u003chtml lang=\\\"en\\\"\\u003e\\n\\u003chead\\u003e\\n    \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n    \\u003ctitle\\u003eKubeflow Log Report\\u003c/title\\u003e\\n    \\u003cstyle\\u003e\\n        body {\\n            font-family: 'Segoe UI', sans-serif;\\n            background-color: #f9f9f9;\\n            color: #333;\\n            padding: 2em;\\n            line-height: 1.6;\\n        }\\n        h1 {\\n            color: #004080;\\n            text-align: center;\\n        }\\n        .namespace {\\n            margin-bottom: 2em;\\n            border: 1px solid #ccc;\\n            padding: 1em;\\n            background-color: #fff;\\n            box-shadow: 0 2px 5px rgba(0,0,0,0.1);\\n            border-radius: 8px;\\n        }\\n        .pod-name {\\n            font-weight: bold;\\n            margin-top: 1em;\\n            font-size: 1.2em;\\n            color: #2a5885;\\n        }\\n        .section-title {\\n            margin-top: 1em;\\n            color: #555;\\n            font-size: 1em;\\n            border-bottom: 1px solid #ddd;\\n        }\\n        pre {\\n            background-color: #272822;\\n            color: #f8f8f2;\\n            padding: 1em;\\n            overflow-x: auto;\\n            white-space: pre-wrap;\\n            border-radius: 4px;\\n        }\\n        .events {\\n            background-color: #fffaf0;\\n            border-left: 5px solid #ffa500;\\n            padding: 1em;\\n            margin-top: 1em;\\n        }\\n    \\u003c/style\\u003e\\n\\u003c/head\\u003e\\n\\u003cbody\\u003e\\n\\u003ch1\\u003eKubeflow Pod Diagnostics Report\\u003c/h1\\u003e\\n\\\"\\\"\\\")\\n\\n        for entry in report_data:\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003ePod: {entry[\\\"pod\\\"]} (Namespace: {entry[\\\"namespace\\\"]})\\u003c/div\\u003e\\\\n')\\n            if entry.get(\\\"describe\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eDescribe\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"describe\\\"]}\\u003c/pre\\u003e\\\\n')\\n            if entry.get(\\\"logs\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eLogs\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"logs\\\"]}\\u003c/pre\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003eEvents in Namespace: {ns}\\u003c/div\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"events\\\"\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/div\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"143"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","39b9d1ea-c847-4bba-ac72-2975413170e4","--dag_execution_id","143","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n    report_data = []\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # 모든 Pod 가져오기\\n    pods_output = subprocess.getoutput(\\n        \\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\"\\n    )\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    for line in pod_lines:\\n        if not line.strip():\\n            continue\\n        try:\\n            namespace, pod_name = line.split()\\n        except ValueError:\\n            continue\\n\\n        describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n        logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n        matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n        matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n        if matched_describe or matched_logs:\\n            report_data.append({\\n                \\\"namespace\\\": namespace,\\n                \\\"pod\\\": pod_name,\\n                \\\"describe\\\": matched_describe,\\n                \\\"logs\\\": matched_logs\\n            })\\n\\n        if namespace not in seen_namespaces:\\n            events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n            matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n            if matched_events:\\n                namespace_events[namespace] = matched_events\\n            seen_namespaces.add(namespace)\\n\\n    # HTML 작성\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(\\\"\\\"\\\"\\n\\u003c!DOCTYPE html\\u003e\\n\\u003chtml lang=\\\"en\\\"\\u003e\\n\\u003chead\\u003e\\n    \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n    \\u003ctitle\\u003eKubeflow Log Report\\u003c/title\\u003e\\n    \\u003cstyle\\u003e\\n        body {\\n            font-family: 'Segoe UI', sans-serif;\\n            background-color: #f9f9f9;\\n            color: #333;\\n            padding: 2em;\\n            line-height: 1.6;\\n        }\\n        h1 {\\n            color: #004080;\\n            text-align: center;\\n        }\\n        .namespace {\\n            margin-bottom: 2em;\\n            border: 1px solid #ccc;\\n            padding: 1em;\\n            background-color: #fff;\\n            box-shadow: 0 2px 5px rgba(0,0,0,0.1);\\n            border-radius: 8px;\\n        }\\n        .pod-name {\\n            font-weight: bold;\\n            margin-top: 1em;\\n            font-size: 1.2em;\\n            color: #2a5885;\\n        }\\n        .section-title {\\n            margin-top: 1em;\\n            color: #555;\\n            font-size: 1em;\\n            border-bottom: 1px solid #ddd;\\n        }\\n        pre {\\n            background-color: #272822;\\n            color: #f8f8f2;\\n            padding: 1em;\\n            overflow-x: auto;\\n            white-space: pre-wrap;\\n            border-radius: 4px;\\n        }\\n        .events {\\n            background-color: #fffaf0;\\n            border-left: 5px solid #ffa500;\\n            padding: 1em;\\n            margin-top: 1em;\\n        }\\n    \\u003c/style\\u003e\\n\\u003c/head\\u003e\\n\\u003cbody\\u003e\\n\\u003ch1\\u003eKubeflow Pod Diagnostics Report\\u003c/h1\\u003e\\n\\\"\\\"\\\")\\n\\n        for entry in report_data:\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003ePod: {entry[\\\"pod\\\"]} (Namespace: {entry[\\\"namespace\\\"]})\\u003c/div\\u003e\\\\n')\\n            if entry.get(\\\"describe\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eDescribe\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"describe\\\"]}\\u003c/pre\\u003e\\\\n')\\n            if entry.get(\\\"logs\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eLogs\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"logs\\\"]}\\u003c/pre\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003eEvents in Namespace: {ns}\\u003c/div\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"events\\\"\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/div\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-xk99f/2025/05/08/pod-log-to-github-xk99f-system-container-driver-537797304"}}}

Logs (filtered)

time="2025-05-08T06:18:04.913Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-xk99f-system-container-impl-2909349134 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"39b9d1ea-c847-4bba-ac72-2975413170e4\",\"--execution_id\",\"144\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/39b9d1ea-c847-4bba-ac72-2975413170e4/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n    report_data = []\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # 모든 Pod 가져오기\\n    pods_output = subprocess.getoutput(\\n        \\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\"\\n    )\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    for line in pod_lines:\\n        if not line.strip():\\n            continue\\n        try:\\n            namespace, pod_name = line.split()\\n        except ValueError:\\n            continue\\n\\n        describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n        logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n        matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n        matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n        if matched_describe or matched_logs:\\n            report_data.append({\\n                \\\"namespace\\\": namespace,\\n                \\\"pod\\\": pod_name,\\n                \\\"describe\\\": matched_describe,\\n                \\\"logs\\\": matched_logs\\n            })\\n\\n        if namespace not in seen_namespaces:\\n            events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n            matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n            if matched_events:\\n                namespace_events[namespace] = matched_events\\n            seen_namespaces.add(namespace)\\n\\n    # HTML 작성\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(\\\"\\\"\\\"\\n\\u003c!DOCTYPE html\\u003e\\n\\u003chtml lang=\\\"en\\\"\\u003e\\n\\u003chead\\u003e\\n    \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n    \\u003ctitle\\u003eKubeflow Log Report\\u003c/title\\u003e\\n    \\u003cstyle\\u003e\\n        body {\\n            font-family: 'Segoe UI', sans-serif;\\n            background-color: #f9f9f9;\\n            color: #333;\\n            padding: 2em;\\n            line-height: 1.6;\\n        }\\n        h1 {\\n            color: #004080;\\n            text-align: center;\\n        }\\n        .namespace {\\n            margin-bottom: 2em;\\n            border: 1px solid #ccc;\\n            padding: 1em;\\n            background-color: #fff;\\n            box-shadow: 0 2px 5px rgba(0,0,0,0.1);\\n            border-radius: 8px;\\n        }\\n        .pod-name {\\n            font-weight: bold;\\n            margin-top: 1em;\\n            font-size: 1.2em;\\n            color: #2a5885;\\n        }\\n        .section-title {\\n            margin-top: 1em;\\n            color: #555;\\n            font-size: 1em;\\n            border-bottom: 1px solid #ddd;\\n        }\\n        pre {\\n            background-color: #272822;\\n            color: #f8f8f2;\\n            padding: 1em;\\n            overflow-x: auto;\\n            white-space: pre-wrap;\\n            border-radius: 4px;\\n        }\\n        .events {\\n            background-color: #fffaf0;\\n            border-left: 5px solid #ffa500;\\n            padding: 1em;\\n            margin-top: 1em;\\n        }\\n    \\u003c/style\\u003e\\n\\u003c/head\\u003e\\n\\u003cbody\\u003e\\n\\u003ch1\\u003eKubeflow Pod Diagnostics Report\\u003c/h1\\u003e\\n\\\"\\\"\\\")\\n\\n        for entry in report_data:\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003ePod: {entry[\\\"pod\\\"]} (Namespace: {entry[\\\"namespace\\\"]})\\u003c/div\\u003e\\\\n')\\n            if entry.get(\\\"describe\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eDescribe\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"describe\\\"]}\\u003c/pre\\u003e\\\\n')\\n            if entry.get(\\\"logs\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eLogs\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"logs\\\"]}\\u003c/pre\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003eEvents in Namespace: {ns}\\u003c/div\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"events\\\"\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/div\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-xk99f/2025/05/08/pod-log-to-github-xk99f-system-container-impl-2909349134"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"39b9d1ea-c847-4bba-ac72-2975413170e4\",\"--execution_id\",\"144\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/39b9d1ea-c847-4bba-ac72-2975413170e4/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n    report_data = []\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # 모든 Pod 가져오기\\n    pods_output = subprocess.getoutput(\\n        \\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\"\\n    )\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    for line in pod_lines:\\n        if not line.strip():\\n            continue\\n        try:\\n            namespace, pod_name = line.split()\\n        except ValueError:\\n            continue\\n\\n        describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n        logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n        matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n        matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n        if matched_describe or matched_logs:\\n            report_data.append({\\n                \\\"namespace\\\": namespace,\\n                \\\"pod\\\": pod_name,\\n                \\\"describe\\\": matched_describe,\\n                \\\"logs\\\": matched_logs\\n            })\\n\\n        if namespace not in seen_namespaces:\\n            events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n            matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n            if matched_events:\\n                namespace_events[namespace] = matched_events\\n            seen_namespaces.add(namespace)\\n\\n    # HTML 작성\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(\\\"\\\"\\\"\\n\\u003c!DOCTYPE html\\u003e\\n\\u003chtml lang=\\\"en\\\"\\u003e\\n\\u003chead\\u003e\\n    \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n    \\u003ctitle\\u003eKubeflow Log Report\\u003c/title\\u003e\\n    \\u003cstyle\\u003e\\n        body {\\n            font-family: 'Segoe UI', sans-serif;\\n            background-color: #f9f9f9;\\n            color: #333;\\n            padding: 2em;\\n            line-height: 1.6;\\n        }\\n        h1 {\\n            color: #004080;\\n            text-align: center;\\n        }\\n        .namespace {\\n            margin-bottom: 2em;\\n            border: 1px solid #ccc;\\n            padding: 1em;\\n            background-color: #fff;\\n            box-shadow: 0 2px 5px rgba(0,0,0,0.1);\\n            border-radius: 8px;\\n        }\\n        .pod-name {\\n            font-weight: bold;\\n            margin-top: 1em;\\n            font-size: 1.2em;\\n            color: #2a5885;\\n        }\\n        .section-title {\\n            margin-top: 1em;\\n            color: #555;\\n            font-size: 1em;\\n            border-bottom: 1px solid #ddd;\\n        }\\n        pre {\\n            background-color: #272822;\\n            color: #f8f8f2;\\n            padding: 1em;\\n            overflow-x: auto;\\n            white-space: pre-wrap;\\n            border-radius: 4px;\\n        }\\n        .events {\\n            background-color: #fffaf0;\\n            border-left: 5px solid #ffa500;\\n            padding: 1em;\\n            margin-top: 1em;\\n        }\\n    \\u003c/style\\u003e\\n\\u003c/head\\u003e\\n\\u003cbody\\u003e\\n\\u003ch1\\u003eKubeflow Pod Diagnostics Report\\u003c/h1\\u003e\\n\\\"\\\"\\\")\\n\\n        for entry in report_data:\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003ePod: {entry[\\\"pod\\\"]} (Namespace: {entry[\\\"namespace\\\"]})\\u003c/div\\u003e\\\\n')\\n            if entry.get(\\\"describe\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eDescribe\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"describe\\\"]}\\u003c/pre\\u003e\\\\n')\\n            if entry.get(\\\"logs\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eLogs\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"logs\\\"]}\\u003c/pre\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003eEvents in Namespace: {ns}\\u003c/div\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"events\\\"\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/div\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"39b9d1ea-c847-4bba-ac72-2975413170e4\",\"--execution_id\",\"144\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/39b9d1ea-c847-4bba-ac72-2975413170e4/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n    report_data = []\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # 모든 Pod 가져오기\\n    pods_output = subprocess.getoutput(\\n        \\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\"\\n    )\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    for line in pod_lines:\\n        if not line.strip():\\n            continue\\n        try:\\n            namespace, pod_name = line.split()\\n        except ValueError:\\n            continue\\n\\n        describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n        logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n        matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n        matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n        if matched_describe or matched_logs:\\n            report_data.append({\\n                \\\"namespace\\\": namespace,\\n                \\\"pod\\\": pod_name,\\n                \\\"describe\\\": matched_describe,\\n                \\\"logs\\\": matched_logs\\n            })\\n\\n        if namespace not in seen_namespaces:\\n            events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n            matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n            if matched_events:\\n                namespace_events[namespace] = matched_events\\n            seen_namespaces.add(namespace)\\n\\n    # HTML 작성\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(\\\"\\\"\\\"\\n\\u003c!DOCTYPE html\\u003e\\n\\u003chtml lang=\\\"en\\\"\\u003e\\n\\u003chead\\u003e\\n    \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n    \\u003ctitle\\u003eKubeflow Log Report\\u003c/title\\u003e\\n    \\u003cstyle\\u003e\\n        body {\\n            font-family: 'Segoe UI', sans-serif;\\n            background-color: #f9f9f9;\\n            color: #333;\\n            padding: 2em;\\n            line-height: 1.6;\\n        }\\n        h1 {\\n            color: #004080;\\n            text-align: center;\\n        }\\n        .namespace {\\n            margin-bottom: 2em;\\n            border: 1px solid #ccc;\\n            padding: 1em;\\n            background-color: #fff;\\n            box-shadow: 0 2px 5px rgba(0,0,0,0.1);\\n            border-radius: 8px;\\n        }\\n        .pod-name {\\n            font-weight: bold;\\n            margin-top: 1em;\\n            font-size: 1.2em;\\n            color: #2a5885;\\n        }\\n        .section-title {\\n            margin-top: 1em;\\n            color: #555;\\n            font-size: 1em;\\n            border-bottom: 1px solid #ddd;\\n        }\\n        pre {\\n            background-color: #272822;\\n            color: #f8f8f2;\\n            padding: 1em;\\n            overflow-x: auto;\\n            white-space: pre-wrap;\\n            border-radius: 4px;\\n        }\\n        .events {\\n            background-color: #fffaf0;\\n            border-left: 5px solid #ffa500;\\n            padding: 1em;\\n            margin-top: 1em;\\n        }\\n    \\u003c/style\\u003e\\n\\u003c/head\\u003e\\n\\u003cbody\\u003e\\n\\u003ch1\\u003eKubeflow Pod Diagnostics Report\\u003c/h1\\u003e\\n\\\"\\\"\\\")\\n\\n        for entry in report_data:\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003ePod: {entry[\\\"pod\\\"]} (Namespace: {entry[\\\"namespace\\\"]})\\u003c/div\\u003e\\\\n')\\n            if entry.get(\\\"describe\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eDescribe\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"describe\\\"]}\\u003c/pre\\u003e\\\\n')\\n            if entry.get(\\\"logs\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eLogs\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"logs\\\"]}\\u003c/pre\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003eEvents in Namespace: {ns}\\u003c/div\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"events\\\"\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/div\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-xk99f/2025/05/08/pod-log-to-github-xk99f-system-container-impl-2909349134"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"39b9d1ea-c847-4bba-ac72-2975413170e4\",\"--execution_id\",\"144\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/39b9d1ea-c847-4bba-ac72-2975413170e4/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n    report_data = []\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # 모든 Pod 가져오기\\n    pods_output = subprocess.getoutput(\\n        \\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\"\\n    )\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    for line in pod_lines:\\n        if not line.strip():\\n            continue\\n        try:\\n            namespace, pod_name = line.split()\\n        except ValueError:\\n            continue\\n\\n        describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n        logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n        matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n        matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n        if matched_describe or matched_logs:\\n            report_data.append({\\n                \\\"namespace\\\": namespace,\\n                \\\"pod\\\": pod_name,\\n                \\\"describe\\\": matched_describe,\\n                \\\"logs\\\": matched_logs\\n            })\\n\\n        if namespace not in seen_namespaces:\\n            events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n            matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n            if matched_events:\\n                namespace_events[namespace] = matched_events\\n            seen_namespaces.add(namespace)\\n\\n    # HTML 작성\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(\\\"\\\"\\\"\\n\\u003c!DOCTYPE html\\u003e\\n\\u003chtml lang=\\\"en\\\"\\u003e\\n\\u003chead\\u003e\\n    \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n    \\u003ctitle\\u003eKubeflow Log Report\\u003c/title\\u003e\\n    \\u003cstyle\\u003e\\n        body {\\n            font-family: 'Segoe UI', sans-serif;\\n            background-color: #f9f9f9;\\n            color: #333;\\n            padding: 2em;\\n            line-height: 1.6;\\n        }\\n        h1 {\\n            color: #004080;\\n            text-align: center;\\n        }\\n        .namespace {\\n            margin-bottom: 2em;\\n            border: 1px solid #ccc;\\n            padding: 1em;\\n            background-color: #fff;\\n            box-shadow: 0 2px 5px rgba(0,0,0,0.1);\\n            border-radius: 8px;\\n        }\\n        .pod-name {\\n            font-weight: bold;\\n            margin-top: 1em;\\n            font-size: 1.2em;\\n            color: #2a5885;\\n        }\\n        .section-title {\\n            margin-top: 1em;\\n            color: #555;\\n            font-size: 1em;\\n            border-bottom: 1px solid #ddd;\\n        }\\n        pre {\\n            background-color: #272822;\\n            color: #f8f8f2;\\n            padding: 1em;\\n            overflow-x: auto;\\n            white-space: pre-wrap;\\n            border-radius: 4px;\\n        }\\n        .events {\\n            background-color: #fffaf0;\\n            border-left: 5px solid #ffa500;\\n            padding: 1em;\\n            margin-top: 1em;\\n        }\\n    \\u003c/style\\u003e\\n\\u003c/head\\u003e\\n\\u003cbody\\u003e\\n\\u003ch1\\u003eKubeflow Pod Diagnostics Report\\u003c/h1\\u003e\\n\\\"\\\"\\\")\\n\\n        for entry in report_data:\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003ePod: {entry[\\\"pod\\\"]} (Namespace: {entry[\\\"namespace\\\"]})\\u003c/div\\u003e\\\\n')\\n            if entry.get(\\\"describe\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eDescribe\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"describe\\\"]}\\u003c/pre\\u003e\\\\n')\\n            if entry.get(\\\"logs\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eLogs\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"logs\\\"]}\\u003c/pre\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003eEvents in Namespace: {ns}\\u003c/div\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"events\\\"\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/div\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"39b9d1ea-c847-4bba-ac72-2975413170e4\",\"--execution_id\",\"144\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/39b9d1ea-c847-4bba-ac72-2975413170e4/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n    report_data = []\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # 모든 Pod 가져오기\\n    pods_output = subprocess.getoutput(\\n        \\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\"\\n    )\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    for line in pod_lines:\\n        if not line.strip():\\n            continue\\n        try:\\n            namespace, pod_name = line.split()\\n        except ValueError:\\n            continue\\n\\n        describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n        logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n        matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n        matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n        if matched_describe or matched_logs:\\n            report_data.append({\\n                \\\"namespace\\\": namespace,\\n                \\\"pod\\\": pod_name,\\n                \\\"describe\\\": matched_describe,\\n                \\\"logs\\\": matched_logs\\n            })\\n\\n        if namespace not in seen_namespaces:\\n            events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n            matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n            if matched_events:\\n                namespace_events[namespace] = matched_events\\n            seen_namespaces.add(namespace)\\n\\n    # HTML 작성\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(\\\"\\\"\\\"\\n\\u003c!DOCTYPE html\\u003e\\n\\u003chtml lang=\\\"en\\\"\\u003e\\n\\u003chead\\u003e\\n    \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n    \\u003ctitle\\u003eKubeflow Log Report\\u003c/title\\u003e\\n    \\u003cstyle\\u003e\\n        body {\\n            font-family: 'Segoe UI', sans-serif;\\n            background-color: #f9f9f9;\\n            color: #333;\\n            padding: 2em;\\n            line-height: 1.6;\\n        }\\n        h1 {\\n            color: #004080;\\n            text-align: center;\\n        }\\n        .namespace {\\n            margin-bottom: 2em;\\n            border: 1px solid #ccc;\\n            padding: 1em;\\n            background-color: #fff;\\n            box-shadow: 0 2px 5px rgba(0,0,0,0.1);\\n            border-radius: 8px;\\n        }\\n        .pod-name {\\n            font-weight: bold;\\n            margin-top: 1em;\\n            font-size: 1.2em;\\n            color: #2a5885;\\n        }\\n        .section-title {\\n            margin-top: 1em;\\n            color: #555;\\n            font-size: 1em;\\n            border-bottom: 1px solid #ddd;\\n        }\\n        pre {\\n            background-color: #272822;\\n            color: #f8f8f2;\\n            padding: 1em;\\n            overflow-x: auto;\\n            white-space: pre-wrap;\\n            border-radius: 4px;\\n        }\\n        .events {\\n            background-color: #fffaf0;\\n            border-left: 5px solid #ffa500;\\n            padding: 1em;\\n            margin-top: 1em;\\n        }\\n    \\u003c/style\\u003e\\n\\u003c/head\\u003e\\n\\u003cbody\\u003e\\n\\u003ch1\\u003eKubeflow Pod Diagnostics Report\\u003c/h1\\u003e\\n\\\"\\\"\\\")\\n\\n        for entry in report_data:\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003ePod: {entry[\\\"pod\\\"]} (Namespace: {entry[\\\"namespace\\\"]})\\u003c/div\\u003e\\\\n')\\n            if entry.get(\\\"describe\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eDescribe\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"describe\\\"]}\\u003c/pre\\u003e\\\\n')\\n            if entry.get(\\\"logs\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eLogs\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"logs\\\"]}\\u003c/pre\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003eEvents in Namespace: {ns}\\u003c/div\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"events\\\"\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/div\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-xk99f/2025/05/08/pod-log-to-github-xk99f-system-container-impl-2909349134"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"39b9d1ea-c847-4bba-ac72-2975413170e4\",\"--execution_id\",\"144\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/39b9d1ea-c847-4bba-ac72-2975413170e4/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n    report_data = []\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # 모든 Pod 가져오기\\n    pods_output = subprocess.getoutput(\\n        \\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\"\\n    )\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    for line in pod_lines:\\n        if not line.strip():\\n            continue\\n        try:\\n            namespace, pod_name = line.split()\\n        except ValueError:\\n            continue\\n\\n        describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n        logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n        matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n        matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n        if matched_describe or matched_logs:\\n            report_data.append({\\n                \\\"namespace\\\": namespace,\\n                \\\"pod\\\": pod_name,\\n                \\\"describe\\\": matched_describe,\\n                \\\"logs\\\": matched_logs\\n            })\\n\\n        if namespace not in seen_namespaces:\\n            events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n            matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n            if matched_events:\\n                namespace_events[namespace] = matched_events\\n            seen_namespaces.add(namespace)\\n\\n    # HTML 작성\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(\\\"\\\"\\\"\\n\\u003c!DOCTYPE html\\u003e\\n\\u003chtml lang=\\\"en\\\"\\u003e\\n\\u003chead\\u003e\\n    \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n    \\u003ctitle\\u003eKubeflow Log Report\\u003c/title\\u003e\\n    \\u003cstyle\\u003e\\n        body {\\n            font-family: 'Segoe UI', sans-serif;\\n            background-color: #f9f9f9;\\n            color: #333;\\n            padding: 2em;\\n            line-height: 1.6;\\n        }\\n        h1 {\\n            color: #004080;\\n            text-align: center;\\n        }\\n        .namespace {\\n            margin-bottom: 2em;\\n            border: 1px solid #ccc;\\n            padding: 1em;\\n            background-color: #fff;\\n            box-shadow: 0 2px 5px rgba(0,0,0,0.1);\\n            border-radius: 8px;\\n        }\\n        .pod-name {\\n            font-weight: bold;\\n            margin-top: 1em;\\n            font-size: 1.2em;\\n            color: #2a5885;\\n        }\\n        .section-title {\\n            margin-top: 1em;\\n            color: #555;\\n            font-size: 1em;\\n            border-bottom: 1px solid #ddd;\\n        }\\n        pre {\\n            background-color: #272822;\\n            color: #f8f8f2;\\n            padding: 1em;\\n            overflow-x: auto;\\n            white-space: pre-wrap;\\n            border-radius: 4px;\\n        }\\n        .events {\\n            background-color: #fffaf0;\\n            border-left: 5px solid #ffa500;\\n            padding: 1em;\\n            margin-top: 1em;\\n        }\\n    \\u003c/style\\u003e\\n\\u003c/head\\u003e\\n\\u003cbody\\u003e\\n\\u003ch1\\u003eKubeflow Pod Diagnostics Report\\u003c/h1\\u003e\\n\\\"\\\"\\\")\\n\\n        for entry in report_data:\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003ePod: {entry[\\\"pod\\\"]} (Namespace: {entry[\\\"namespace\\\"]})\\u003c/div\\u003e\\\\n')\\n            if entry.get(\\\"describe\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eDescribe\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"describe\\\"]}\\u003c/pre\\u003e\\\\n')\\n            if entry.get(\\\"logs\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eLogs\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"logs\\\"]}\\u003c/pre\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003eEvents in Namespace: {ns}\\u003c/div\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"events\\\"\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/div\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
              except ValueError:
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"39b9d1ea-c847-4bba-ac72-2975413170e4\",\"--execution_id\",\"144\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/39b9d1ea-c847-4bba-ac72-2975413170e4/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n    report_data = []\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # 모든 Pod 가져오기\\n    pods_output = subprocess.getoutput(\\n        \\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\"\\n    )\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    for line in pod_lines:\\n        if not line.strip():\\n            continue\\n        try:\\n            namespace, pod_name = line.split()\\n        except ValueError:\\n            continue\\n\\n        describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n        logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n        matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n        matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n        if matched_describe or matched_logs:\\n            report_data.append({\\n                \\\"namespace\\\": namespace,\\n                \\\"pod\\\": pod_name,\\n                \\\"describe\\\": matched_describe,\\n                \\\"logs\\\": matched_logs\\n            })\\n\\n        if namespace not in seen_namespaces:\\n            events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n            matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n            if matched_events:\\n                namespace_events[namespace] = matched_events\\n            seen_namespaces.add(namespace)\\n\\n    # HTML 작성\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(\\\"\\\"\\\"\\n\\u003c!DOCTYPE html\\u003e\\n\\u003chtml lang=\\\"en\\\"\\u003e\\n\\u003chead\\u003e\\n    \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n    \\u003ctitle\\u003eKubeflow Log Report\\u003c/title\\u003e\\n    \\u003cstyle\\u003e\\n        body {\\n            font-family: 'Segoe UI', sans-serif;\\n            background-color: #f9f9f9;\\n            color: #333;\\n            padding: 2em;\\n            line-height: 1.6;\\n        }\\n        h1 {\\n            color: #004080;\\n            text-align: center;\\n        }\\n        .namespace {\\n            margin-bottom: 2em;\\n            border: 1px solid #ccc;\\n            padding: 1em;\\n            background-color: #fff;\\n            box-shadow: 0 2px 5px rgba(0,0,0,0.1);\\n            border-radius: 8px;\\n        }\\n        .pod-name {\\n            font-weight: bold;\\n            margin-top: 1em;\\n            font-size: 1.2em;\\n            color: #2a5885;\\n        }\\n        .section-title {\\n            margin-top: 1em;\\n            color: #555;\\n            font-size: 1em;\\n            border-bottom: 1px solid #ddd;\\n        }\\n        pre {\\n            background-color: #272822;\\n            color: #f8f8f2;\\n            padding: 1em;\\n            overflow-x: auto;\\n            white-space: pre-wrap;\\n            border-radius: 4px;\\n        }\\n        .events {\\n            background-color: #fffaf0;\\n            border-left: 5px solid #ffa500;\\n            padding: 1em;\\n            margin-top: 1em;\\n        }\\n    \\u003c/style\\u003e\\n\\u003c/head\\u003e\\n\\u003cbody\\u003e\\n\\u003ch1\\u003eKubeflow Pod Diagnostics Report\\u003c/h1\\u003e\\n\\\"\\\"\\\")\\n\\n        for entry in report_data:\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003ePod: {entry[\\\"pod\\\"]} (Namespace: {entry[\\\"namespace\\\"]})\\u003c/div\\u003e\\\\n')\\n            if entry.get(\\\"describe\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eDescribe\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"describe\\\"]}\\u003c/pre\\u003e\\\\n')\\n            if entry.get(\\\"logs\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eLogs\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"logs\\\"]}\\u003c/pre\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003eEvents in Namespace: {ns}\\u003c/div\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"events\\\"\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/div\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-xk99f/2025/05/08/pod-log-to-github-xk99f-system-container-impl-2909349134"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"39b9d1ea-c847-4bba-ac72-2975413170e4\",\"--execution_id\",\"144\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/39b9d1ea-c847-4bba-ac72-2975413170e4/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n    report_data = []\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # 모든 Pod 가져오기\\n    pods_output = subprocess.getoutput(\\n        \\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\"\\n    )\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    for line in pod_lines:\\n        if not line.strip():\\n            continue\\n        try:\\n            namespace, pod_name = line.split()\\n        except ValueError:\\n            continue\\n\\n        describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n        logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n        matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n        matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n        if matched_describe or matched_logs:\\n            report_data.append({\\n                \\\"namespace\\\": namespace,\\n                \\\"pod\\\": pod_name,\\n                \\\"describe\\\": matched_describe,\\n                \\\"logs\\\": matched_logs\\n            })\\n\\n        if namespace not in seen_namespaces:\\n            events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n            matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n            if matched_events:\\n                namespace_events[namespace] = matched_events\\n            seen_namespaces.add(namespace)\\n\\n    # HTML 작성\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(\\\"\\\"\\\"\\n\\u003c!DOCTYPE html\\u003e\\n\\u003chtml lang=\\\"en\\\"\\u003e\\n\\u003chead\\u003e\\n    \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n    \\u003ctitle\\u003eKubeflow Log Report\\u003c/title\\u003e\\n    \\u003cstyle\\u003e\\n        body {\\n            font-family: 'Segoe UI', sans-serif;\\n            background-color: #f9f9f9;\\n            color: #333;\\n            padding: 2em;\\n            line-height: 1.6;\\n        }\\n        h1 {\\n            color: #004080;\\n            text-align: center;\\n        }\\n        .namespace {\\n            margin-bottom: 2em;\\n            border: 1px solid #ccc;\\n            padding: 1em;\\n            background-color: #fff;\\n            box-shadow: 0 2px 5px rgba(0,0,0,0.1);\\n            border-radius: 8px;\\n        }\\n        .pod-name {\\n            font-weight: bold;\\n            margin-top: 1em;\\n            font-size: 1.2em;\\n            color: #2a5885;\\n        }\\n        .section-title {\\n            margin-top: 1em;\\n            color: #555;\\n            font-size: 1em;\\n            border-bottom: 1px solid #ddd;\\n        }\\n        pre {\\n            background-color: #272822;\\n            color: #f8f8f2;\\n            padding: 1em;\\n            overflow-x: auto;\\n            white-space: pre-wrap;\\n            border-radius: 4px;\\n        }\\n        .events {\\n            background-color: #fffaf0;\\n            border-left: 5px solid #ffa500;\\n            padding: 1em;\\n            margin-top: 1em;\\n        }\\n    \\u003c/style\\u003e\\n\\u003c/head\\u003e\\n\\u003cbody\\u003e\\n\\u003ch1\\u003eKubeflow Pod Diagnostics Report\\u003c/h1\\u003e\\n\\\"\\\"\\\")\\n\\n        for entry in report_data:\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003ePod: {entry[\\\"pod\\\"]} (Namespace: {entry[\\\"namespace\\\"]})\\u003c/div\\u003e\\\\n')\\n            if entry.get(\\\"describe\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eDescribe\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"describe\\\"]}\\u003c/pre\\u003e\\\\n')\\n            if entry.get(\\\"logs\\\"):\\n                f.write(f'\\u003cdiv class=\\\"section-title\\\"\\u003eLogs\\u003c/div\\u003e\\\\n\\u003cpre\\u003e{entry[\\\"logs\\\"]}\\u003c/pre\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f'\\u003cdiv class=\\\"namespace\\\"\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"pod-name\\\"\\u003eEvents in Namespace: {ns}\\u003c/div\\u003e\\\\n')\\n            f.write(f'\\u003cdiv class=\\\"events\\\"\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/div\\u003e\\\\n')\\n            f.write('\\u003c/div\\u003e\\\\n')\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}

Logs (filtered)

time="2025-05-08T06:19:28.323Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T06:19:29.231Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-xk99f-system-container-impl-3339243367 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"39b9d1ea-c847-4bba-ac72-2975413170e4\",\"--execution_id\",\"145\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"25\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/39b9d1ea-c847-4bba-ac72-2975413170e4/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-xk99f/2025/05/08/pod-log-to-github-xk99f-system-container-impl-3339243367"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"39b9d1ea-c847-4bba-ac72-2975413170e4\",\"--execution_id\",\"145\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"25\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/39b9d1ea-c847-4bba-ac72-2975413170e4/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"39b9d1ea-c847-4bba-ac72-2975413170e4\",\"--execution_id\",\"145\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"25\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/39b9d1ea-c847-4bba-ac72-2975413170e4/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-xk99f/2025/05/08/pod-log-to-github-xk99f-system-container-impl-3339243367"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"39b9d1ea-c847-4bba-ac72-2975413170e4\",\"--execution_id\",\"145\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"25\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/39b9d1ea-c847-4bba-ac72-2975413170e4/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"39b9d1ea-c847-4bba-ac72-2975413170e4\",\"--execution_id\",\"145\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"25\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/39b9d1ea-c847-4bba-ac72-2975413170e4/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-xk99f/2025/05/08/pod-log-to-github-xk99f-system-container-impl-3339243367"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"39b9d1ea-c847-4bba-ac72-2975413170e4\",\"--execution_id\",\"145\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"25\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/39b9d1ea-c847-4bba-ac72-2975413170e4/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
              raise FileNotFoundError(f"{report_path} does not exist")
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"39b9d1ea-c847-4bba-ac72-2975413170e4\",\"--execution_id\",\"145\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"25\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/39b9d1ea-c847-4bba-ac72-2975413170e4/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-xk99f/2025/05/08/pod-log-to-github-xk99f-system-container-impl-3339243367"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"39b9d1ea-c847-4bba-ac72-2975413170e4\",\"--execution_id\",\"145\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"25\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/39b9d1ea-c847-4bba-ac72-2975413170e4/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}

Logs (filtered)

time="2025-05-08T06:20:02.325Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T06:20:03.251Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-xk99f-system-dag-driver-165966120 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T06:17:54.821Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-z2xx6-system-container-driver-1293330074 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"147"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","1e12f20c-0f3e-4a06-8369-f77d6da1428f","--dag_execution_id","147","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-z2xx6/2025/05/08/pod-log-to-github-z2xx6-system-container-driver-1293330074"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"147"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","1e12f20c-0f3e-4a06-8369-f77d6da1428f","--dag_execution_id","147","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-z2xx6/2025/05/08/pod-log-to-github-z2xx6-system-container-driver-1293330074"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str\n):\n\n    import subprocess\n    import shutil\n    import os\n    from datetime import datetime\n\n    os.environ[\"GIT_ASKPASS\"] = \"echo\"\n\n    report_path = report_file.path\n    print(\"Checking if report exists at:\", report_path)\n    if not os.path.exists(report_path):\n        raise FileNotFoundError(f\"{report_path} does not exist\")\n\n    # HTML 파일 이름 만들기\n    timestamp = datetime.now().strftime(\"report-%Y%m%d-%H%M%S.html\")\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n    report_target_path = os.path.join(repo_dir, timestamp)\n\n    # Git clone\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    shutil.copyfile(report_path, report_target_path)\n\n    # Git config\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    # Git commit \u0026 push\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", f\"Add {timestamp}\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(f\"Report committed and pushed to GitHub as {timestamp}.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"147"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","1e12f20c-0f3e-4a06-8369-f77d6da1428f","--dag_execution_id","147","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-z2xx6/2025/05/08/pod-log-to-github-z2xx6-system-container-driver-1293330074"}}}

Logs (filtered)

time="2025-05-08T06:33:34.741Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-z2xx6-system-container-driver-231423181 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # 주요 키워드 목록\\n    describe_keywords = [\\\"CrashLoopBackOff\\\", \\\"OOMKilled\\\", \\\"Error\\\", \\\"Failed\\\", \\\"Back-off\\\"]\\n    log_keywords = [\\\"Traceback\\\", \\\"panic\\\", \\\"exception\\\", \\\"fail\\\", \\\"error\\\"]\\n    event_keywords = [\\\"Failed\\\", \\\"BackOff\\\", \\\"Unhealthy\\\", \\\"Killing\\\", \\\"Evicted\\\"]\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    def match_keywords(lines, keywords):\\n        return \\\"\\\\n\\\".join([line for line in lines if any(k in line for k in keywords)])\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        # HTML header\\n        f.write(\\\"\\\"\\\"\\n        \\u003chtml\\u003e\\n        \\u003chead\\u003e\\n            \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n            \\u003cstyle\\u003e\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\n                h1 { color: #2c3e50; }\\n                h2 { color: #34495e; }\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\n                section { margin-bottom: 30px; }\\n            \\u003c/style\\u003e\\n        \\u003c/head\\u003e\\n        \\u003cbody\\u003e\\n        \\u003ch1\\u003eKubernetes 장애 점검 리포트\\u003c/h1\\u003e\\n        \\\"\\\"\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe_output = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            log_output = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003ePod: {pod_name} (Namespace: {namespace})\\u003c/h2\\u003e\\\")\\n                if matched_describe:\\n                    f.write(\\\"\\u003ch3\\u003eDescribe (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_describe + \\\"\\u003c/pre\\u003e\\\")\\n                if matched_logs:\\n                    f.write(\\\"\\u003ch3\\u003eLogs (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_logs + \\\"\\u003c/pre\\u003e\\\")\\n                f.write(\\\"\\u003c/section\\u003e\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events_output = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003eEvents in Namespace: {ns}\\u003c/h2\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/section\\u003e\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"147"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","1e12f20c-0f3e-4a06-8369-f77d6da1428f","--dag_execution_id","147","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # 주요 키워드 목록\\n    describe_keywords = [\\\"CrashLoopBackOff\\\", \\\"OOMKilled\\\", \\\"Error\\\", \\\"Failed\\\", \\\"Back-off\\\"]\\n    log_keywords = [\\\"Traceback\\\", \\\"panic\\\", \\\"exception\\\", \\\"fail\\\", \\\"error\\\"]\\n    event_keywords = [\\\"Failed\\\", \\\"BackOff\\\", \\\"Unhealthy\\\", \\\"Killing\\\", \\\"Evicted\\\"]\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    def match_keywords(lines, keywords):\\n        return \\\"\\\\n\\\".join([line for line in lines if any(k in line for k in keywords)])\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        # HTML header\\n        f.write(\\\"\\\"\\\"\\n        \\u003chtml\\u003e\\n        \\u003chead\\u003e\\n            \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n            \\u003cstyle\\u003e\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\n                h1 { color: #2c3e50; }\\n                h2 { color: #34495e; }\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\n                section { margin-bottom: 30px; }\\n            \\u003c/style\\u003e\\n        \\u003c/head\\u003e\\n        \\u003cbody\\u003e\\n        \\u003ch1\\u003eKubernetes 장애 점검 리포트\\u003c/h1\\u003e\\n        \\\"\\\"\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe_output = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            log_output = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003ePod: {pod_name} (Namespace: {namespace})\\u003c/h2\\u003e\\\")\\n                if matched_describe:\\n                    f.write(\\\"\\u003ch3\\u003eDescribe (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_describe + \\\"\\u003c/pre\\u003e\\\")\\n                if matched_logs:\\n                    f.write(\\\"\\u003ch3\\u003eLogs (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_logs + \\\"\\u003c/pre\\u003e\\\")\\n                f.write(\\\"\\u003c/section\\u003e\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events_output = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003eEvents in Namespace: {ns}\\u003c/h2\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/section\\u003e\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-z2xx6/2025/05/08/pod-log-to-github-z2xx6-system-container-driver-231423181"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # 주요 키워드 목록\\n    describe_keywords = [\\\"CrashLoopBackOff\\\", \\\"OOMKilled\\\", \\\"Error\\\", \\\"Failed\\\", \\\"Back-off\\\"]\\n    log_keywords = [\\\"Traceback\\\", \\\"panic\\\", \\\"exception\\\", \\\"fail\\\", \\\"error\\\"]\\n    event_keywords = [\\\"Failed\\\", \\\"BackOff\\\", \\\"Unhealthy\\\", \\\"Killing\\\", \\\"Evicted\\\"]\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    def match_keywords(lines, keywords):\\n        return \\\"\\\\n\\\".join([line for line in lines if any(k in line for k in keywords)])\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        # HTML header\\n        f.write(\\\"\\\"\\\"\\n        \\u003chtml\\u003e\\n        \\u003chead\\u003e\\n            \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n            \\u003cstyle\\u003e\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\n                h1 { color: #2c3e50; }\\n                h2 { color: #34495e; }\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\n                section { margin-bottom: 30px; }\\n            \\u003c/style\\u003e\\n        \\u003c/head\\u003e\\n        \\u003cbody\\u003e\\n        \\u003ch1\\u003eKubernetes 장애 점검 리포트\\u003c/h1\\u003e\\n        \\\"\\\"\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe_output = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            log_output = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003ePod: {pod_name} (Namespace: {namespace})\\u003c/h2\\u003e\\\")\\n                if matched_describe:\\n                    f.write(\\\"\\u003ch3\\u003eDescribe (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_describe + \\\"\\u003c/pre\\u003e\\\")\\n                if matched_logs:\\n                    f.write(\\\"\\u003ch3\\u003eLogs (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_logs + \\\"\\u003c/pre\\u003e\\\")\\n                f.write(\\\"\\u003c/section\\u003e\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events_output = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003eEvents in Namespace: {ns}\\u003c/h2\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/section\\u003e\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"147"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","1e12f20c-0f3e-4a06-8369-f77d6da1428f","--dag_execution_id","147","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # 주요 키워드 목록\\n    describe_keywords = [\\\"CrashLoopBackOff\\\", \\\"OOMKilled\\\", \\\"Error\\\", \\\"Failed\\\", \\\"Back-off\\\"]\\n    log_keywords = [\\\"Traceback\\\", \\\"panic\\\", \\\"exception\\\", \\\"fail\\\", \\\"error\\\"]\\n    event_keywords = [\\\"Failed\\\", \\\"BackOff\\\", \\\"Unhealthy\\\", \\\"Killing\\\", \\\"Evicted\\\"]\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    def match_keywords(lines, keywords):\\n        return \\\"\\\\n\\\".join([line for line in lines if any(k in line for k in keywords)])\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        # HTML header\\n        f.write(\\\"\\\"\\\"\\n        \\u003chtml\\u003e\\n        \\u003chead\\u003e\\n            \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n            \\u003cstyle\\u003e\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\n                h1 { color: #2c3e50; }\\n                h2 { color: #34495e; }\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\n                section { margin-bottom: 30px; }\\n            \\u003c/style\\u003e\\n        \\u003c/head\\u003e\\n        \\u003cbody\\u003e\\n        \\u003ch1\\u003eKubernetes 장애 점검 리포트\\u003c/h1\\u003e\\n        \\\"\\\"\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe_output = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            log_output = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003ePod: {pod_name} (Namespace: {namespace})\\u003c/h2\\u003e\\\")\\n                if matched_describe:\\n                    f.write(\\\"\\u003ch3\\u003eDescribe (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_describe + \\\"\\u003c/pre\\u003e\\\")\\n                if matched_logs:\\n                    f.write(\\\"\\u003ch3\\u003eLogs (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_logs + \\\"\\u003c/pre\\u003e\\\")\\n                f.write(\\\"\\u003c/section\\u003e\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events_output = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003eEvents in Namespace: {ns}\\u003c/h2\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/section\\u003e\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-z2xx6/2025/05/08/pod-log-to-github-z2xx6-system-container-driver-231423181"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","collect_filtered_logs"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef collect_filtered_logs(output_report: Output[Dataset]):\n    import subprocess\n    import os\n\n    tmp_report_path = \"/tmp/report.html\"\n    seen_namespaces = set()\n    namespace_events = {}\n\n    # 주요 키워드 목록\n    describe_keywords = [\"CrashLoopBackOff\", \"OOMKilled\", \"Error\", \"Failed\", \"Back-off\"]\n    log_keywords = [\"Traceback\", \"panic\", \"exception\", \"fail\", \"error\"]\n    event_keywords = [\"Failed\", \"BackOff\", \"Unhealthy\", \"Killing\", \"Evicted\"]\n\n    # kubectl 설치\n    subprocess.run([\"apt-get\", \"update\", \"-y\"], check=True)\n    subprocess.run([\n        \"curl\", \"-LO\", \"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\"\n    ], check=True)\n    subprocess.run([\"chmod\", \"+x\", \"kubectl\"], check=True)\n    subprocess.run([\"mv\", \"kubectl\", \"/usr/local/bin/\"], check=True)\n\n    # Get all pods\n    pods_output = subprocess.getoutput(\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\")\n    pod_lines = pods_output.strip().split(\"\\n\")\n\n    def match_keywords(lines, keywords):\n        return \"\\n\".join([line for line in lines if any(k in line for k in keywords)])\n\n    with open(tmp_report_path, \"w\") as f:\n        # HTML header\n        f.write(\"\"\"\n        \u003chtml\u003e\n        \u003chead\u003e\n            \u003cmeta charset=\"UTF-8\"\u003e\n            \u003cstyle\u003e\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\n                h1 { color: #2c3e50; }\n                h2 { color: #34495e; }\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\n                section { margin-bottom: 30px; }\n            \u003c/style\u003e\n        \u003c/head\u003e\n        \u003cbody\u003e\n        \u003ch1\u003eKubernetes 장애 점검 리포트\u003c/h1\u003e\n        \"\"\")\n\n        for line in pod_lines:\n            if not line.strip():\n                continue\n            try:\n                namespace, pod_name = line.split()\n            except ValueError:\n                continue\n\n            describe_output = subprocess.getoutput(f\"kubectl describe pod {pod_name} -n {namespace}\")\n            log_output = subprocess.getoutput(f\"kubectl logs {pod_name} -n {namespace}\")\n\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\n\n            if matched_describe or matched_logs:\n                f.write(f\"\u003csection\u003e\u003ch2\u003ePod: {pod_name} (Namespace: {namespace})\u003c/h2\u003e\")\n                if matched_describe:\n                    f.write(\"\u003ch3\u003eDescribe (filtered)\u003c/h3\u003e\u003cpre\u003e\" + matched_describe + \"\u003c/pre\u003e\")\n                if matched_logs:\n                    f.write(\"\u003ch3\u003eLogs (filtered)\u003c/h3\u003e\u003cpre\u003e\" + matched_logs + \"\u003c/pre\u003e\")\n                f.write(\"\u003c/section\u003e\")\n\n            if namespace not in seen_namespaces:\n                events_output = subprocess.getoutput(f\"kubectl get events -n {namespace}\")\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\n                if matched_events:\n                    namespace_events[namespace] = matched_events\n                seen_namespaces.add(namespace)\n\n        for ns, events in namespace_events.items():\n            f.write(f\"\u003csection\u003e\u003ch2\u003eEvents in Namespace: {ns}\u003c/h2\u003e\u003cpre\u003e{events}\u003c/pre\u003e\u003c/section\u003e\")\n\n        f.write(\"\u003c/body\u003e\u003c/html\u003e\")\n\n    subprocess.run([\"cp\", tmp_report_path, output_report.path], check=True)\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # 주요 키워드 목록\\n    describe_keywords = [\\\"CrashLoopBackOff\\\", \\\"OOMKilled\\\", \\\"Error\\\", \\\"Failed\\\", \\\"Back-off\\\"]\\n    log_keywords = [\\\"Traceback\\\", \\\"panic\\\", \\\"exception\\\", \\\"fail\\\", \\\"error\\\"]\\n    event_keywords = [\\\"Failed\\\", \\\"BackOff\\\", \\\"Unhealthy\\\", \\\"Killing\\\", \\\"Evicted\\\"]\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    def match_keywords(lines, keywords):\\n        return \\\"\\\\n\\\".join([line for line in lines if any(k in line for k in keywords)])\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        # HTML header\\n        f.write(\\\"\\\"\\\"\\n        \\u003chtml\\u003e\\n        \\u003chead\\u003e\\n            \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n            \\u003cstyle\\u003e\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\n                h1 { color: #2c3e50; }\\n                h2 { color: #34495e; }\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\n                section { margin-bottom: 30px; }\\n            \\u003c/style\\u003e\\n        \\u003c/head\\u003e\\n        \\u003cbody\\u003e\\n        \\u003ch1\\u003eKubernetes 장애 점검 리포트\\u003c/h1\\u003e\\n        \\\"\\\"\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe_output = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            log_output = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003ePod: {pod_name} (Namespace: {namespace})\\u003c/h2\\u003e\\\")\\n                if matched_describe:\\n                    f.write(\\\"\\u003ch3\\u003eDescribe (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_describe + \\\"\\u003c/pre\\u003e\\\")\\n                if matched_logs:\\n                    f.write(\\\"\\u003ch3\\u003eLogs (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_logs + \\\"\\u003c/pre\\u003e\\\")\\n                f.write(\\\"\\u003c/section\\u003e\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events_output = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003eEvents in Namespace: {ns}\\u003c/h2\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/section\\u003e\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"147"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","1e12f20c-0f3e-4a06-8369-f77d6da1428f","--dag_execution_id","147","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # 주요 키워드 목록\\n    describe_keywords = [\\\"CrashLoopBackOff\\\", \\\"OOMKilled\\\", \\\"Error\\\", \\\"Failed\\\", \\\"Back-off\\\"]\\n    log_keywords = [\\\"Traceback\\\", \\\"panic\\\", \\\"exception\\\", \\\"fail\\\", \\\"error\\\"]\\n    event_keywords = [\\\"Failed\\\", \\\"BackOff\\\", \\\"Unhealthy\\\", \\\"Killing\\\", \\\"Evicted\\\"]\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    def match_keywords(lines, keywords):\\n        return \\\"\\\\n\\\".join([line for line in lines if any(k in line for k in keywords)])\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        # HTML header\\n        f.write(\\\"\\\"\\\"\\n        \\u003chtml\\u003e\\n        \\u003chead\\u003e\\n            \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n            \\u003cstyle\\u003e\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\n                h1 { color: #2c3e50; }\\n                h2 { color: #34495e; }\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\n                section { margin-bottom: 30px; }\\n            \\u003c/style\\u003e\\n        \\u003c/head\\u003e\\n        \\u003cbody\\u003e\\n        \\u003ch1\\u003eKubernetes 장애 점검 리포트\\u003c/h1\\u003e\\n        \\\"\\\"\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe_output = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            log_output = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003ePod: {pod_name} (Namespace: {namespace})\\u003c/h2\\u003e\\\")\\n                if matched_describe:\\n                    f.write(\\\"\\u003ch3\\u003eDescribe (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_describe + \\\"\\u003c/pre\\u003e\\\")\\n                if matched_logs:\\n                    f.write(\\\"\\u003ch3\\u003eLogs (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_logs + \\\"\\u003c/pre\\u003e\\\")\\n                f.write(\\\"\\u003c/section\\u003e\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events_output = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003eEvents in Namespace: {ns}\\u003c/h2\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/section\\u003e\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-z2xx6/2025/05/08/pod-log-to-github-z2xx6-system-container-driver-231423181"}}}

Logs (filtered)

    "\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef collect_filtered_logs(output_report: Output[Dataset]):\n    import subprocess\n    import os\n\n    tmp_report_path = \"/tmp/report.html\"\n    seen_namespaces = set()\n    namespace_events = {}\n\n    # 주요 키워드 목록\n    describe_keywords = [\"CrashLoopBackOff\", \"OOMKilled\", \"Error\", \"Failed\", \"Back-off\"]\n    log_keywords = [\"Traceback\", \"panic\", \"exception\", \"fail\", \"error\"]\n    event_keywords = [\"Failed\", \"BackOff\", \"Unhealthy\", \"Killing\", \"Evicted\"]\n\n    # kubectl 설치\n    subprocess.run([\"apt-get\", \"update\", \"-y\"], check=True)\n    subprocess.run([\n        \"curl\", \"-LO\", \"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\"\n    ], check=True)\n    subprocess.run([\"chmod\", \"+x\", \"kubectl\"], check=True)\n    subprocess.run([\"mv\", \"kubectl\", \"/usr/local/bin/\"], check=True)\n\n    # Get all pods\n    pods_output = subprocess.getoutput(\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\")\n    pod_lines = pods_output.strip().split(\"\\n\")\n\n    def match_keywords(lines, keywords):\n        return \"\\n\".join([line for line in lines if any(k in line for k in keywords)])\n\n    with open(tmp_report_path, \"w\") as f:\n        # HTML header\n        f.write(\"\"\"\n        \u003chtml\u003e\n        \u003chead\u003e\n            \u003cmeta charset=\"UTF-8\"\u003e\n            \u003cstyle\u003e\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\n                h1 { color: #2c3e50; }\n                h2 { color: #34495e; }\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\n                section { margin-bottom: 30px; }\n            \u003c/style\u003e\n        \u003c/head\u003e\n        \u003cbody\u003e\n        \u003ch1\u003eKubernetes 장애 점검 리포트\u003c/h1\u003e\n        \"\"\")\n\n        for line in pod_lines:\n            if not line.strip():\n                continue\n            try:\n                namespace, pod_name = line.split()\n            except ValueError:\n                continue\n\n            describe_output = subprocess.getoutput(f\"kubectl describe pod {pod_name} -n {namespace}\")\n            log_output = subprocess.getoutput(f\"kubectl logs {pod_name} -n {namespace}\")\n\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\n\n            if matched_describe or matched_logs:\n                f.write(f\"\u003csection\u003e\u003ch2\u003ePod: {pod_name} (Namespace: {namespace})\u003c/h2\u003e\")\n                if matched_describe:\n                    f.write(\"\u003ch3\u003eDescribe (filtered)\u003c/h3\u003e\u003cpre\u003e\" + matched_describe + \"\u003c/pre\u003e\")\n                if matched_logs:\n                    f.write(\"\u003ch3\u003eLogs (filtered)\u003c/h3\u003e\u003cpre\u003e\" + matched_logs + \"\u003c/pre\u003e\")\n                f.write(\"\u003c/section\u003e\")\n\n            if namespace not in seen_namespaces:\n                events_output = subprocess.getoutput(f\"kubectl get events -n {namespace}\")\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\n                if matched_events:\n                    namespace_events[namespace] = matched_events\n                seen_namespaces.add(namespace)\n\n        for ns, events in namespace_events.items():\n            f.write(f\"\u003csection\u003e\u003ch2\u003eEvents in Namespace: {ns}\u003c/h2\u003e\u003cpre\u003e{events}\u003c/pre\u003e\u003c/section\u003e\")\n\n        f.write(\"\u003c/body\u003e\u003c/html\u003e\")\n\n    subprocess.run([\"cp\", tmp_report_path, output_report.path], check=True)\n\n"
{"containers":[{"name":"main","image":"python:3.9","command":["/var/run/argo/argoexec","emissary","--","/kfp-launcher/launch","--pipeline_name","pod-log-to-github","--run_id","1e12f20c-0f3e-4a06-8369-f77d6da1428f","--execution_id","148","--executor_input","{\"inputs\":{}, \"outputs\":{\"artifacts\":{\"output_report\":{\"artifacts\":[{\"type\":{\"schemaTitle\":\"system.Dataset\", \"schemaVersion\":\"0.0.1\"}, \"uri\":\"minio://mlpipeline/v2/artifacts/pod-log-to-github/1e12f20c-0f3e-4a06-8369-f77d6da1428f/collect-filtered-logs/output_report\"}]}}, \"outputFile\":\"/tmp/kfp_outputs/output_metadata.json\"}}","--component_spec","{\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\", \"schemaVersion\":\"0.0.1\"}}}}, \"executorLabel\":\"exec-collect-filtered-logs\"}","--pod_name","$(KFP_POD_NAME)","--pod_uid","$(KFP_POD_UID)","--mlmd_server_address","$(METADATA_GRPC_SERVICE_HOST)","--mlmd_server_port","$(METADATA_GRPC_SERVICE_PORT)","--"],"args":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef collect_filtered_logs(output_report: Output[Dataset]):\n    import subprocess\n    import os\n\n    tmp_report_path = \"/tmp/report.html\"\n    seen_namespaces = set()\n    namespace_events = {}\n\n    # 주요 키워드 목록\n    describe_keywords = [\"CrashLoopBackOff\", \"OOMKilled\", \"Error\", \"Failed\", \"Back-off\"]\n    log_keywords = [\"Traceback\", \"panic\", \"exception\", \"fail\", \"error\"]\n    event_keywords = [\"Failed\", \"BackOff\", \"Unhealthy\", \"Killing\", \"Evicted\"]\n\n    # kubectl 설치\n    subprocess.run([\"apt-get\", \"update\", \"-y\"], check=True)\n    subprocess.run([\n        \"curl\", \"-LO\", \"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\"\n    ], check=True)\n    subprocess.run([\"chmod\", \"+x\", \"kubectl\"], check=True)\n    subprocess.run([\"mv\", \"kubectl\", \"/usr/local/bin/\"], check=True)\n\n    # Get all pods\n    pods_output = subprocess.getoutput(\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\")\n    pod_lines = pods_output.strip().split(\"\\n\")\n\n    def match_keywords(lines, keywords):\n        return \"\\n\".join([line for line in lines if any(k in line for k in keywords)])\n\n    with open(tmp_report_path, \"w\") as f:\n        # HTML header\n        f.write(\"\"\"\n        \u003chtml\u003e\n        \u003chead\u003e\n            \u003cmeta charset=\"UTF-8\"\u003e\n            \u003cstyle\u003e\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\n                h1 { color: #2c3e50; }\n                h2 { color: #34495e; }\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\n                section { margin-bottom: 30px; }\n            \u003c/style\u003e\n        \u003c/head\u003e\n        \u003cbody\u003e\n        \u003ch1\u003eKubernetes 장애 점검 리포트\u003c/h1\u003e\n        \"\"\")\n\n        for line in pod_lines:\n            if not line.strip():\n                continue\n            try:\n                namespace, pod_name = line.split()\n            except ValueError:\n                continue\n\n            describe_output = subprocess.getoutput(f\"kubectl describe pod {pod_name} -n {namespace}\")\n            log_output = subprocess.getoutput(f\"kubectl logs {pod_name} -n {namespace}\")\n\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\n\n            if matched_describe or matched_logs:\n                f.write(f\"\u003csection\u003e\u003ch2\u003ePod: {pod_name} (Namespace: {namespace})\u003c/h2\u003e\")\n                if matched_describe:\n                    f.write(\"\u003ch3\u003eDescribe (filtered)\u003c/h3\u003e\u003cpre\u003e\" + matched_describe + \"\u003c/pre\u003e\")\n                if matched_logs:\n                    f.write(\"\u003ch3\u003eLogs (filtered)\u003c/h3\u003e\u003cpre\u003e\" + matched_logs + \"\u003c/pre\u003e\")\n                f.write(\"\u003c/section\u003e\")\n\n            if namespace not in seen_namespaces:\n                events_output = subprocess.getoutput(f\"kubectl get events -n {namespace}\")\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\n                if matched_events:\n                    namespace_events[namespace] = matched_events\n                seen_namespaces.add(namespace)\n\n        for ns, events in namespace_events.items():\n            f.write(f\"\u003csection\u003e\u003ch2\u003eEvents in Namespace: {ns}\u003c/h2\u003e\u003cpre\u003e{events}\u003c/pre\u003e\u003c/section\u003e\")\n\n        f.write(\"\u003c/body\u003e\u003c/html\u003e\")\n\n    subprocess.run([\"cp\", tmp_report_path, output_report.path], check=True)\n\n","--executor_input","{{$}}","--function_to_execute","collect_filtered_logs"],"resources":{}}]}
time="2025-05-08T06:31:55.519Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-z2xx6-system-container-impl-1861443604 (Namespace: kubeflow-user-example-com)

Describe (filtered)

Status:       Failed
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--execution_id\",\"149\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"26\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/1e12f20c-0f3e-4a06-8369-f77d6da1428f/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-z2xx6/2025/05/08/pod-log-to-github-z2xx6-system-container-impl-1861443604"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--execution_id\",\"149\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"26\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/1e12f20c-0f3e-4a06-8369-f77d6da1428f/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--execution_id\",\"149\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"26\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/1e12f20c-0f3e-4a06-8369-f77d6da1428f/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-z2xx6/2025/05/08/pod-log-to-github-z2xx6-system-container-impl-1861443604"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--execution_id\",\"149\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"26\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/1e12f20c-0f3e-4a06-8369-f77d6da1428f/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--execution_id\",\"149\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"26\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/1e12f20c-0f3e-4a06-8369-f77d6da1428f/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-z2xx6/2025/05/08/pod-log-to-github-z2xx6-system-container-impl-1861443604"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--execution_id\",\"149\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"26\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/1e12f20c-0f3e-4a06-8369-f77d6da1428f/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
              raise FileNotFoundError(f"{report_path} does not exist")
      Reason:       Error
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--execution_id\",\"149\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"26\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/1e12f20c-0f3e-4a06-8369-f77d6da1428f/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-z2xx6/2025/05/08/pod-log-to-github-z2xx6-system-container-impl-1861443604"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--execution_id\",\"149\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"26\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/1e12f20c-0f3e-4a06-8369-f77d6da1428f/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}

Logs (filtered)

remote: error: GH013: Repository rule violations found for refs/heads/report.        
error: failed to push some refs to 'https://github.com/Rosa1026/Kubeflow.git'
Traceback (most recent call last):
F0508 06:33:55.672334      23 main.go:49] failed to execute component: exit status 1
time="2025-05-08T06:33:55.948Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T06:33:56.889Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-z2xx6-system-container-impl-3251966491 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--execution_id\",\"148\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/1e12f20c-0f3e-4a06-8369-f77d6da1428f/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # 주요 키워드 목록\\n    describe_keywords = [\\\"CrashLoopBackOff\\\", \\\"OOMKilled\\\", \\\"Error\\\", \\\"Failed\\\", \\\"Back-off\\\"]\\n    log_keywords = [\\\"Traceback\\\", \\\"panic\\\", \\\"exception\\\", \\\"fail\\\", \\\"error\\\"]\\n    event_keywords = [\\\"Failed\\\", \\\"BackOff\\\", \\\"Unhealthy\\\", \\\"Killing\\\", \\\"Evicted\\\"]\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    def match_keywords(lines, keywords):\\n        return \\\"\\\\n\\\".join([line for line in lines if any(k in line for k in keywords)])\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        # HTML header\\n        f.write(\\\"\\\"\\\"\\n        \\u003chtml\\u003e\\n        \\u003chead\\u003e\\n            \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n            \\u003cstyle\\u003e\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\n                h1 { color: #2c3e50; }\\n                h2 { color: #34495e; }\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\n                section { margin-bottom: 30px; }\\n            \\u003c/style\\u003e\\n        \\u003c/head\\u003e\\n        \\u003cbody\\u003e\\n        \\u003ch1\\u003eKubernetes 장애 점검 리포트\\u003c/h1\\u003e\\n        \\\"\\\"\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe_output = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            log_output = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003ePod: {pod_name} (Namespace: {namespace})\\u003c/h2\\u003e\\\")\\n                if matched_describe:\\n                    f.write(\\\"\\u003ch3\\u003eDescribe (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_describe + \\\"\\u003c/pre\\u003e\\\")\\n                if matched_logs:\\n                    f.write(\\\"\\u003ch3\\u003eLogs (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_logs + \\\"\\u003c/pre\\u003e\\\")\\n                f.write(\\\"\\u003c/section\\u003e\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events_output = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003eEvents in Namespace: {ns}\\u003c/h2\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/section\\u003e\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-z2xx6/2025/05/08/pod-log-to-github-z2xx6-system-container-impl-3251966491"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--execution_id\",\"148\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/1e12f20c-0f3e-4a06-8369-f77d6da1428f/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # 주요 키워드 목록\\n    describe_keywords = [\\\"CrashLoopBackOff\\\", \\\"OOMKilled\\\", \\\"Error\\\", \\\"Failed\\\", \\\"Back-off\\\"]\\n    log_keywords = [\\\"Traceback\\\", \\\"panic\\\", \\\"exception\\\", \\\"fail\\\", \\\"error\\\"]\\n    event_keywords = [\\\"Failed\\\", \\\"BackOff\\\", \\\"Unhealthy\\\", \\\"Killing\\\", \\\"Evicted\\\"]\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    def match_keywords(lines, keywords):\\n        return \\\"\\\\n\\\".join([line for line in lines if any(k in line for k in keywords)])\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        # HTML header\\n        f.write(\\\"\\\"\\\"\\n        \\u003chtml\\u003e\\n        \\u003chead\\u003e\\n            \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n            \\u003cstyle\\u003e\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\n                h1 { color: #2c3e50; }\\n                h2 { color: #34495e; }\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\n                section { margin-bottom: 30px; }\\n            \\u003c/style\\u003e\\n        \\u003c/head\\u003e\\n        \\u003cbody\\u003e\\n        \\u003ch1\\u003eKubernetes 장애 점검 리포트\\u003c/h1\\u003e\\n        \\\"\\\"\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe_output = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            log_output = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003ePod: {pod_name} (Namespace: {namespace})\\u003c/h2\\u003e\\\")\\n                if matched_describe:\\n                    f.write(\\\"\\u003ch3\\u003eDescribe (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_describe + \\\"\\u003c/pre\\u003e\\\")\\n                if matched_logs:\\n                    f.write(\\\"\\u003ch3\\u003eLogs (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_logs + \\\"\\u003c/pre\\u003e\\\")\\n                f.write(\\\"\\u003c/section\\u003e\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events_output = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003eEvents in Namespace: {ns}\\u003c/h2\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/section\\u003e\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--execution_id\",\"148\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/1e12f20c-0f3e-4a06-8369-f77d6da1428f/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # 주요 키워드 목록\\n    describe_keywords = [\\\"CrashLoopBackOff\\\", \\\"OOMKilled\\\", \\\"Error\\\", \\\"Failed\\\", \\\"Back-off\\\"]\\n    log_keywords = [\\\"Traceback\\\", \\\"panic\\\", \\\"exception\\\", \\\"fail\\\", \\\"error\\\"]\\n    event_keywords = [\\\"Failed\\\", \\\"BackOff\\\", \\\"Unhealthy\\\", \\\"Killing\\\", \\\"Evicted\\\"]\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    def match_keywords(lines, keywords):\\n        return \\\"\\\\n\\\".join([line for line in lines if any(k in line for k in keywords)])\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        # HTML header\\n        f.write(\\\"\\\"\\\"\\n        \\u003chtml\\u003e\\n        \\u003chead\\u003e\\n            \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n            \\u003cstyle\\u003e\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\n                h1 { color: #2c3e50; }\\n                h2 { color: #34495e; }\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\n                section { margin-bottom: 30px; }\\n            \\u003c/style\\u003e\\n        \\u003c/head\\u003e\\n        \\u003cbody\\u003e\\n        \\u003ch1\\u003eKubernetes 장애 점검 리포트\\u003c/h1\\u003e\\n        \\\"\\\"\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe_output = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            log_output = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003ePod: {pod_name} (Namespace: {namespace})\\u003c/h2\\u003e\\\")\\n                if matched_describe:\\n                    f.write(\\\"\\u003ch3\\u003eDescribe (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_describe + \\\"\\u003c/pre\\u003e\\\")\\n                if matched_logs:\\n                    f.write(\\\"\\u003ch3\\u003eLogs (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_logs + \\\"\\u003c/pre\\u003e\\\")\\n                f.write(\\\"\\u003c/section\\u003e\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events_output = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003eEvents in Namespace: {ns}\\u003c/h2\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/section\\u003e\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-z2xx6/2025/05/08/pod-log-to-github-z2xx6-system-container-impl-3251966491"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--execution_id\",\"148\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/1e12f20c-0f3e-4a06-8369-f77d6da1428f/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # 주요 키워드 목록\\n    describe_keywords = [\\\"CrashLoopBackOff\\\", \\\"OOMKilled\\\", \\\"Error\\\", \\\"Failed\\\", \\\"Back-off\\\"]\\n    log_keywords = [\\\"Traceback\\\", \\\"panic\\\", \\\"exception\\\", \\\"fail\\\", \\\"error\\\"]\\n    event_keywords = [\\\"Failed\\\", \\\"BackOff\\\", \\\"Unhealthy\\\", \\\"Killing\\\", \\\"Evicted\\\"]\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    def match_keywords(lines, keywords):\\n        return \\\"\\\\n\\\".join([line for line in lines if any(k in line for k in keywords)])\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        # HTML header\\n        f.write(\\\"\\\"\\\"\\n        \\u003chtml\\u003e\\n        \\u003chead\\u003e\\n            \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n            \\u003cstyle\\u003e\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\n                h1 { color: #2c3e50; }\\n                h2 { color: #34495e; }\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\n                section { margin-bottom: 30px; }\\n            \\u003c/style\\u003e\\n        \\u003c/head\\u003e\\n        \\u003cbody\\u003e\\n        \\u003ch1\\u003eKubernetes 장애 점검 리포트\\u003c/h1\\u003e\\n        \\\"\\\"\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe_output = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            log_output = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003ePod: {pod_name} (Namespace: {namespace})\\u003c/h2\\u003e\\\")\\n                if matched_describe:\\n                    f.write(\\\"\\u003ch3\\u003eDescribe (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_describe + \\\"\\u003c/pre\\u003e\\\")\\n                if matched_logs:\\n                    f.write(\\\"\\u003ch3\\u003eLogs (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_logs + \\\"\\u003c/pre\\u003e\\\")\\n                f.write(\\\"\\u003c/section\\u003e\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events_output = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003eEvents in Namespace: {ns}\\u003c/h2\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/section\\u003e\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--execution_id\",\"148\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/1e12f20c-0f3e-4a06-8369-f77d6da1428f/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # 주요 키워드 목록\\n    describe_keywords = [\\\"CrashLoopBackOff\\\", \\\"OOMKilled\\\", \\\"Error\\\", \\\"Failed\\\", \\\"Back-off\\\"]\\n    log_keywords = [\\\"Traceback\\\", \\\"panic\\\", \\\"exception\\\", \\\"fail\\\", \\\"error\\\"]\\n    event_keywords = [\\\"Failed\\\", \\\"BackOff\\\", \\\"Unhealthy\\\", \\\"Killing\\\", \\\"Evicted\\\"]\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    def match_keywords(lines, keywords):\\n        return \\\"\\\\n\\\".join([line for line in lines if any(k in line for k in keywords)])\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        # HTML header\\n        f.write(\\\"\\\"\\\"\\n        \\u003chtml\\u003e\\n        \\u003chead\\u003e\\n            \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n            \\u003cstyle\\u003e\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\n                h1 { color: #2c3e50; }\\n                h2 { color: #34495e; }\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\n                section { margin-bottom: 30px; }\\n            \\u003c/style\\u003e\\n        \\u003c/head\\u003e\\n        \\u003cbody\\u003e\\n        \\u003ch1\\u003eKubernetes 장애 점검 리포트\\u003c/h1\\u003e\\n        \\\"\\\"\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe_output = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            log_output = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003ePod: {pod_name} (Namespace: {namespace})\\u003c/h2\\u003e\\\")\\n                if matched_describe:\\n                    f.write(\\\"\\u003ch3\\u003eDescribe (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_describe + \\\"\\u003c/pre\\u003e\\\")\\n                if matched_logs:\\n                    f.write(\\\"\\u003ch3\\u003eLogs (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_logs + \\\"\\u003c/pre\\u003e\\\")\\n                f.write(\\\"\\u003c/section\\u003e\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events_output = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003eEvents in Namespace: {ns}\\u003c/h2\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/section\\u003e\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-z2xx6/2025/05/08/pod-log-to-github-z2xx6-system-container-impl-3251966491"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--execution_id\",\"148\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/1e12f20c-0f3e-4a06-8369-f77d6da1428f/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # 주요 키워드 목록\\n    describe_keywords = [\\\"CrashLoopBackOff\\\", \\\"OOMKilled\\\", \\\"Error\\\", \\\"Failed\\\", \\\"Back-off\\\"]\\n    log_keywords = [\\\"Traceback\\\", \\\"panic\\\", \\\"exception\\\", \\\"fail\\\", \\\"error\\\"]\\n    event_keywords = [\\\"Failed\\\", \\\"BackOff\\\", \\\"Unhealthy\\\", \\\"Killing\\\", \\\"Evicted\\\"]\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    def match_keywords(lines, keywords):\\n        return \\\"\\\\n\\\".join([line for line in lines if any(k in line for k in keywords)])\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        # HTML header\\n        f.write(\\\"\\\"\\\"\\n        \\u003chtml\\u003e\\n        \\u003chead\\u003e\\n            \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n            \\u003cstyle\\u003e\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\n                h1 { color: #2c3e50; }\\n                h2 { color: #34495e; }\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\n                section { margin-bottom: 30px; }\\n            \\u003c/style\\u003e\\n        \\u003c/head\\u003e\\n        \\u003cbody\\u003e\\n        \\u003ch1\\u003eKubernetes 장애 점검 리포트\\u003c/h1\\u003e\\n        \\\"\\\"\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe_output = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            log_output = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003ePod: {pod_name} (Namespace: {namespace})\\u003c/h2\\u003e\\\")\\n                if matched_describe:\\n                    f.write(\\\"\\u003ch3\\u003eDescribe (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_describe + \\\"\\u003c/pre\\u003e\\\")\\n                if matched_logs:\\n                    f.write(\\\"\\u003ch3\\u003eLogs (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_logs + \\\"\\u003c/pre\\u003e\\\")\\n                f.write(\\\"\\u003c/section\\u003e\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events_output = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003eEvents in Namespace: {ns}\\u003c/h2\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/section\\u003e\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
          describe_keywords = ["CrashLoopBackOff", "OOMKilled", "Error", "Failed", "Back-off"]
          event_keywords = ["Failed", "BackOff", "Unhealthy", "Killing", "Evicted"]
                  except ValueError:
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--execution_id\",\"148\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/1e12f20c-0f3e-4a06-8369-f77d6da1428f/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # 주요 키워드 목록\\n    describe_keywords = [\\\"CrashLoopBackOff\\\", \\\"OOMKilled\\\", \\\"Error\\\", \\\"Failed\\\", \\\"Back-off\\\"]\\n    log_keywords = [\\\"Traceback\\\", \\\"panic\\\", \\\"exception\\\", \\\"fail\\\", \\\"error\\\"]\\n    event_keywords = [\\\"Failed\\\", \\\"BackOff\\\", \\\"Unhealthy\\\", \\\"Killing\\\", \\\"Evicted\\\"]\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    def match_keywords(lines, keywords):\\n        return \\\"\\\\n\\\".join([line for line in lines if any(k in line for k in keywords)])\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        # HTML header\\n        f.write(\\\"\\\"\\\"\\n        \\u003chtml\\u003e\\n        \\u003chead\\u003e\\n            \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n            \\u003cstyle\\u003e\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\n                h1 { color: #2c3e50; }\\n                h2 { color: #34495e; }\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\n                section { margin-bottom: 30px; }\\n            \\u003c/style\\u003e\\n        \\u003c/head\\u003e\\n        \\u003cbody\\u003e\\n        \\u003ch1\\u003eKubernetes 장애 점검 리포트\\u003c/h1\\u003e\\n        \\\"\\\"\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe_output = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            log_output = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003ePod: {pod_name} (Namespace: {namespace})\\u003c/h2\\u003e\\\")\\n                if matched_describe:\\n                    f.write(\\\"\\u003ch3\\u003eDescribe (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_describe + \\\"\\u003c/pre\\u003e\\\")\\n                if matched_logs:\\n                    f.write(\\\"\\u003ch3\\u003eLogs (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_logs + \\\"\\u003c/pre\\u003e\\\")\\n                f.write(\\\"\\u003c/section\\u003e\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events_output = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003eEvents in Namespace: {ns}\\u003c/h2\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/section\\u003e\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-z2xx6/2025/05/08/pod-log-to-github-z2xx6-system-container-impl-3251966491"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--execution_id\",\"148\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/1e12f20c-0f3e-4a06-8369-f77d6da1428f/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # 주요 키워드 목록\\n    describe_keywords = [\\\"CrashLoopBackOff\\\", \\\"OOMKilled\\\", \\\"Error\\\", \\\"Failed\\\", \\\"Back-off\\\"]\\n    log_keywords = [\\\"Traceback\\\", \\\"panic\\\", \\\"exception\\\", \\\"fail\\\", \\\"error\\\"]\\n    event_keywords = [\\\"Failed\\\", \\\"BackOff\\\", \\\"Unhealthy\\\", \\\"Killing\\\", \\\"Evicted\\\"]\\n\\n    # kubectl 설치\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([\\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    def match_keywords(lines, keywords):\\n        return \\\"\\\\n\\\".join([line for line in lines if any(k in line for k in keywords)])\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        # HTML header\\n        f.write(\\\"\\\"\\\"\\n        \\u003chtml\\u003e\\n        \\u003chead\\u003e\\n            \\u003cmeta charset=\\\"UTF-8\\\"\\u003e\\n            \\u003cstyle\\u003e\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\n                h1 { color: #2c3e50; }\\n                h2 { color: #34495e; }\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\n                section { margin-bottom: 30px; }\\n            \\u003c/style\\u003e\\n        \\u003c/head\\u003e\\n        \\u003cbody\\u003e\\n        \\u003ch1\\u003eKubernetes 장애 점검 리포트\\u003c/h1\\u003e\\n        \\\"\\\"\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe_output = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            log_output = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003ePod: {pod_name} (Namespace: {namespace})\\u003c/h2\\u003e\\\")\\n                if matched_describe:\\n                    f.write(\\\"\\u003ch3\\u003eDescribe (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_describe + \\\"\\u003c/pre\\u003e\\\")\\n                if matched_logs:\\n                    f.write(\\\"\\u003ch3\\u003eLogs (filtered)\\u003c/h3\\u003e\\u003cpre\\u003e\\\" + matched_logs + \\\"\\u003c/pre\\u003e\\\")\\n                f.write(\\\"\\u003c/section\\u003e\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events_output = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003csection\\u003e\\u003ch2\\u003eEvents in Namespace: {ns}\\u003c/h2\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\u003c/section\\u003e\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}

Logs (filtered)

time="2025-05-08T06:33:20.823Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T06:33:21.766Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-z2xx6-system-dag-driver-1009858533 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T06:31:45.321Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-z9lb2-system-container-driver-2064061380 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T07:37:35.624Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-z9lb2-system-container-impl-2128329098 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      Reason:       Error
  Normal  BackOff  4m19s (x5948 over 22h)  kubelet  Back-off pulling image "python3.9"

Logs (filtered)

Error from server (BadRequest): container "main" in pod "pod-log-to-github-z9lb2-system-container-impl-2128329098" is waiting to start: trying and failing to pull image

Pod: pod-log-to-github-z9lb2-system-dag-driver-2179919500 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-07T07:37:25.529Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-zctnf-system-container-driver-1361082799 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"116"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","0ea50e35-0e80-4ff5-af64-1ee79b441f3c","--dag_execution_id","116","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-zctnf/2025/05/08/pod-log-to-github-zctnf-system-container-driver-1361082799"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"116"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","0ea50e35-0e80-4ff5-af64-1ee79b441f3c","--dag_execution_id","116","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-zctnf/2025/05/08/pod-log-to-github-zctnf-system-container-driver-1361082799"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str = \"main\"\n):\n\n    import subprocess\n    import shutil\n    import os\n    from datetime import datetime\n\n    report_path = report_file.path\n    print(\"Checking if report exists at:\", report_path)\n    if not os.path.exists(report_path):\n        raise FileNotFoundError(f\"{report_path} does not exist\")\n\n    # HTML 파일 이름 만들기\n    timestamp = datetime.now().strftime(\"report-%Y%m%d-%H%M%S.html\")\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n    report_target_path = os.path.join(repo_dir, timestamp)\n\n    # Git clone\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    shutil.copyfile(report_path, report_target_path)\n\n    # Git config\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    # Git commit \u0026 push\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", f\"Add {timestamp}\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(f\"Report committed and pushed to GitHub as {timestamp}.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"116"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","0ea50e35-0e80-4ff5-af64-1ee79b441f3c","--dag_execution_id","116","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-zctnf/2025/05/08/pod-log-to-github-zctnf-system-container-driver-1361082799"}}}

Logs (filtered)

time="2025-05-08T05:04:46.423Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-zctnf-system-container-driver-50637130 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"116"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","0ea50e35-0e80-4ff5-af64-1ee79b441f3c","--dag_execution_id","116","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-zctnf/2025/05/08/pod-log-to-github-zctnf-system-container-driver-50637130"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"116"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","0ea50e35-0e80-4ff5-af64-1ee79b441f3c","--dag_execution_id","116","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-zctnf/2025/05/08/pod-log-to-github-zctnf-system-container-driver-50637130"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","collect_filtered_logs"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \"crash\"):\n    import shutil\n    import json\n    import subprocess\n    import os\n\n    tmp_report_path = \"/tmp/report.html\"\n    seen_namespaces = set()\n    namespace_events = {}\n\n    # kubectl 설치 방법 변경\n    subprocess.run([\"apt-get\", \"update\", \"-y\"], check=True)\n    subprocess.run([ \n        \"curl\", \"-LO\", \"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\"\n    ], check=True)\n    subprocess.run([\"chmod\", \"+x\", \"kubectl\"], check=True)\n    subprocess.run([\"mv\", \"kubectl\", \"/usr/local/bin/\"], check=True)\n\n    # Get all pods in all namespaces\n    pods_output = subprocess.getoutput(\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\")\n    pod_lines = pods_output.strip().split(\"\\n\")\n\n    with open(tmp_report_path, \"w\") as f:\n        f.write(f\"\u003chtml\u003e\u003cbody\u003e\u003ch1\u003e장애 복기 리포트 (keyword: `{keyword}`)\u003c/h1\u003e\\n\")\n\n        for line in pod_lines:\n            if not line.strip():\n                continue\n            try:\n                namespace, pod_name = line.split()\n            except ValueError:\n                continue\n\n            describe = subprocess.getoutput(f\"kubectl describe pod {pod_name} -n {namespace}\")\n            logs = subprocess.getoutput(f\"kubectl logs {pod_name} -n {namespace}\")\n\n            matched_describe = \"\\n\".join([l for l in describe.splitlines() if keyword in l])\n            matched_logs = \"\\n\".join([l for l in logs.splitlines() if keyword in l])\n\n            if matched_describe or matched_logs:\n                f.write(f\"\u003ch2\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\u003c/h2\u003e\\n\")\n                if matched_describe:\n                    f.write(f\"\u003ch3\u003eDescribe\u003c/h3\u003e\u003cpre\u003e{matched_describe}\u003c/pre\u003e\\n\")\n                if matched_logs:\n                    f.write(f\"\u003ch3\u003eLogs\u003c/h3\u003e\u003cpre\u003e{matched_logs}\u003c/pre\u003e\\n\")\n\n            if namespace not in seen_namespaces:\n                events = subprocess.getoutput(f\"kubectl get events -n {namespace}\")\n                matched_events = \"\\n\".join([l for l in events.splitlines() if keyword in l])\n                if matched_events:\n                    namespace_events[namespace] = matched_events\n                seen_namespaces.add(namespace)\n\n        # Write filtered events grouped by namespace\n        for ns, events in namespace_events.items():\n            f.write(f\"\u003ch3\u003eEvents in Namespace: `{ns}`\u003c/h3\u003e\u003cpre\u003e{events}\u003c/pre\u003e\\n\")\n\n        f.write(\"\u003c/body\u003e\u003c/html\u003e\")\n\n    subprocess.run([\"cp\", tmp_report_path, output_report.path], check=True)\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"116"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","0ea50e35-0e80-4ff5-af64-1ee79b441f3c","--dag_execution_id","116","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-zctnf/2025/05/08/pod-log-to-github-zctnf-system-container-driver-50637130"}}}

Logs (filtered)

time="2025-05-08T05:03:12.423Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-zctnf-system-container-impl-1076826660 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"0ea50e35-0e80-4ff5-af64-1ee79b441f3c\",\"--execution_id\",\"117\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/0ea50e35-0e80-4ff5-af64-1ee79b441f3c/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-zctnf/2025/05/08/pod-log-to-github-zctnf-system-container-impl-1076826660"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"0ea50e35-0e80-4ff5-af64-1ee79b441f3c\",\"--execution_id\",\"117\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/0ea50e35-0e80-4ff5-af64-1ee79b441f3c/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"0ea50e35-0e80-4ff5-af64-1ee79b441f3c\",\"--execution_id\",\"117\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/0ea50e35-0e80-4ff5-af64-1ee79b441f3c/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-zctnf/2025/05/08/pod-log-to-github-zctnf-system-container-impl-1076826660"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"0ea50e35-0e80-4ff5-af64-1ee79b441f3c\",\"--execution_id\",\"117\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/0ea50e35-0e80-4ff5-af64-1ee79b441f3c/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"0ea50e35-0e80-4ff5-af64-1ee79b441f3c\",\"--execution_id\",\"117\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/0ea50e35-0e80-4ff5-af64-1ee79b441f3c/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-zctnf/2025/05/08/pod-log-to-github-zctnf-system-container-impl-1076826660"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"0ea50e35-0e80-4ff5-af64-1ee79b441f3c\",\"--execution_id\",\"117\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/0ea50e35-0e80-4ff5-af64-1ee79b441f3c/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
                  except ValueError:
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"0ea50e35-0e80-4ff5-af64-1ee79b441f3c\",\"--execution_id\",\"117\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/0ea50e35-0e80-4ff5-af64-1ee79b441f3c/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-zctnf/2025/05/08/pod-log-to-github-zctnf-system-container-impl-1076826660"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"0ea50e35-0e80-4ff5-af64-1ee79b441f3c\",\"--execution_id\",\"117\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/0ea50e35-0e80-4ff5-af64-1ee79b441f3c/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}

Logs (filtered)

time="2025-05-08T05:04:32.612Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T05:04:33.543Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-zctnf-system-container-impl-3813999009 (Namespace: kubeflow-user-example-com)

Describe (filtered)

Status:       Failed
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"0ea50e35-0e80-4ff5-af64-1ee79b441f3c\",\"--execution_id\",\"118\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"16\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/0ea50e35-0e80-4ff5-af64-1ee79b441f3c/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-zctnf/2025/05/08/pod-log-to-github-zctnf-system-container-impl-3813999009"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"0ea50e35-0e80-4ff5-af64-1ee79b441f3c\",\"--execution_id\",\"118\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"16\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/0ea50e35-0e80-4ff5-af64-1ee79b441f3c/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"0ea50e35-0e80-4ff5-af64-1ee79b441f3c\",\"--execution_id\",\"118\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"16\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/0ea50e35-0e80-4ff5-af64-1ee79b441f3c/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-zctnf/2025/05/08/pod-log-to-github-zctnf-system-container-impl-3813999009"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"0ea50e35-0e80-4ff5-af64-1ee79b441f3c\",\"--execution_id\",\"118\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"16\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/0ea50e35-0e80-4ff5-af64-1ee79b441f3c/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"0ea50e35-0e80-4ff5-af64-1ee79b441f3c\",\"--execution_id\",\"118\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"16\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/0ea50e35-0e80-4ff5-af64-1ee79b441f3c/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-zctnf/2025/05/08/pod-log-to-github-zctnf-system-container-impl-3813999009"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"0ea50e35-0e80-4ff5-af64-1ee79b441f3c\",\"--execution_id\",\"118\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"16\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/0ea50e35-0e80-4ff5-af64-1ee79b441f3c/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
              raise FileNotFoundError(f"{report_path} does not exist")
      Reason:       Error
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"0ea50e35-0e80-4ff5-af64-1ee79b441f3c\",\"--execution_id\",\"118\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"16\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/0ea50e35-0e80-4ff5-af64-1ee79b441f3c/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-zctnf/2025/05/08/pod-log-to-github-zctnf-system-container-impl-3813999009"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"0ea50e35-0e80-4ff5-af64-1ee79b441f3c\",\"--execution_id\",\"118\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"16\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/0ea50e35-0e80-4ff5-af64-1ee79b441f3c/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}

Logs (filtered)

Traceback (most recent call last):
F0508 05:05:05.972093      23 main.go:49] failed to execute component: exit status 1
time="2025-05-08T05:05:06.633Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T05:05:07.579Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-zctnf-system-dag-driver-3129824910 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T05:03:02.223Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-zj76w-system-container-driver-2328641866 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"125"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","2e7ffed3-456d-4740-98ac-f89e1c2eb2b9","--dag_execution_id","125","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-zj76w/2025/05/08/pod-log-to-github-zj76w-system-container-driver-2328641866"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"125"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","2e7ffed3-456d-4740-98ac-f89e1c2eb2b9","--dag_execution_id","125","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-zj76w/2025/05/08/pod-log-to-github-zj76w-system-container-driver-2328641866"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","upload_to_github"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"'  \u0026\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef upload_to_github(\n    report_file: Input[Dataset],\n    repo_url: str,\n    github_token: str,\n    branch: str = \"main\"\n):\n\n    import subprocess\n    import shutil\n    import os\n    from datetime import datetime\n\n    os.environ[\"GIT_ASKPASS\"] = \"echo\"\n\n    report_path = report_file.path\n    print(\"Checking if report exists at:\", report_path)\n    if not os.path.exists(report_path):\n        raise FileNotFoundError(f\"{report_path} does not exist\")\n\n    # HTML 파일 이름 만들기\n    timestamp = datetime.now().strftime(\"report-%Y%m%d-%H%M%S.html\")\n    safe_repo_url = repo_url.replace(\"https://\", f\"https://{github_token}@\")\n    repo_dir = \"/tmp/repo\"\n    report_target_path = os.path.join(repo_dir, timestamp)\n\n    # Git clone\n    subprocess.run([\"git\", \"clone\", \"--branch\", branch, safe_repo_url, repo_dir], check=True)\n    shutil.copyfile(report_path, report_target_path)\n\n    # Git config\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.email\", \"ci@example.com\"], check=True)\n    subprocess.run([\"git\", \"-C\", repo_dir, \"config\", \"user.name\", \"CI Bot\"], check=True)\n\n    # Git commit \u0026 push\n    subprocess.run([\"git\", \"-C\", repo_dir, \"add\", \".\"], check=True)\n    result = subprocess.run([\"git\", \"-C\", repo_dir, \"status\", \"--porcelain\"], capture_output=True, text=True)\n\n    if result.stdout.strip():\n        subprocess.run([\"git\", \"-C\", repo_dir, \"commit\", \"-m\", f\"Add {timestamp}\"], check=True)\n        subprocess.run([\"git\", \"-C\", repo_dir, \"push\"], check=True)\n        print(f\"Report committed and pushed to GitHub as {timestamp}.\")\n    else:\n        print(\"No changes to commit. Skipping.\")\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}"},{"name":"task","value":"{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"125"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","2e7ffed3-456d-4740-98ac-f89e1c2eb2b9","--dag_execution_id","125","--component","{\"executorLabel\":\"exec-upload-to-github\",\"inputDefinitions\":{\"artifacts\":{\"report_file\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}},\"parameters\":{\"branch\":{\"defaultValue\":\"main\",\"isOptional\":true,\"parameterType\":\"STRING\"},\"github_token\":{\"parameterType\":\"STRING\"},\"repo_url\":{\"parameterType\":\"STRING\"}}}}","--task","{\"cachingOptions\":{\"enableCache\":true},\"componentRef\":{\"name\":\"comp-upload-to-github\"},\"dependentTasks\":[\"collect-filtered-logs\"],\"inputs\":{\"artifacts\":{\"report_file\":{\"taskOutputArtifact\":{\"outputArtifactKey\":\"output_report\",\"producerTask\":\"collect-filtered-logs\"}}},\"parameters\":{\"branch\":{\"componentInputParameter\":\"branch\"},\"github_token\":{\"componentInputParameter\":\"github_token\"},\"repo_url\":{\"componentInputParameter\":\"repo_url\"}}},\"taskInfo\":{\"name\":\"upload-to-github\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-zj76w/2025/05/08/pod-log-to-github-zj76w-system-container-driver-2328641866"}}}

Logs (filtered)

time="2025-05-08T05:42:53.923Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-zj76w-system-container-driver-878217725 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"125"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","2e7ffed3-456d-4740-98ac-f89e1c2eb2b9","--dag_execution_id","125","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-zj76w/2025/05/08/pod-log-to-github-zj76w-system-container-driver-878217725"}}}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"125"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","2e7ffed3-456d-4740-98ac-f89e1c2eb2b9","--dag_execution_id","125","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-zj76w/2025/05/08/pod-log-to-github-zj76w-system-container-driver-878217725"}}}
      {"args":["--executor_input","{{$}}","--function_to_execute","collect_filtered_logs"],"command":["sh","-c","\nif ! [ -x \"$(command -v pip)\" ]; then\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\nfi\n\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\u003e=3.7.4,\u003c5; python_version\u003c\"3.9\"' \u0026\u0026 \"$0\" \"$@\"\n","sh","-ec","program_path=$(mktemp -d)\n\nprintf \"%s\" \"$0\" \u003e \"$program_path/ephemeral_component.py\"\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \"$program_path/ephemeral_component.py\"                         \"$@\"\n","\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import *\n\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \"crash\"):\n    import shutil\n    import json\n    import subprocess\n    import os\n\n    tmp_report_path = \"/tmp/report.html\"\n    seen_namespaces = set()\n    namespace_events = {}\n\n    # kubectl 설치 방법 변경\n    subprocess.run([\"apt-get\", \"update\", \"-y\"], check=True)\n    subprocess.run([ \n        \"curl\", \"-LO\", \"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\"\n    ], check=True)\n    subprocess.run([\"chmod\", \"+x\", \"kubectl\"], check=True)\n    subprocess.run([\"mv\", \"kubectl\", \"/usr/local/bin/\"], check=True)\n\n    # Get all pods in all namespaces\n    pods_output = subprocess.getoutput(\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\")\n    pod_lines = pods_output.strip().split(\"\\n\")\n\n    with open(tmp_report_path, \"w\") as f:\n        f.write(f\"\u003chtml\u003e\u003cbody\u003e\u003ch1\u003e장애 복기 리포트 (keyword: `{keyword}`)\u003c/h1\u003e\\n\")\n\n        for line in pod_lines:\n            if not line.strip():\n                continue\n            try:\n                namespace, pod_name = line.split()\n            except ValueError:\n                continue\n\n            describe = subprocess.getoutput(f\"kubectl describe pod {pod_name} -n {namespace}\")\n            logs = subprocess.getoutput(f\"kubectl logs {pod_name} -n {namespace}\")\n\n            matched_describe = \"\\n\".join([l for l in describe.splitlines() if keyword in l])\n            matched_logs = \"\\n\".join([l for l in logs.splitlines() if keyword in l])\n\n            if matched_describe or matched_logs:\n                f.write(f\"\u003ch2\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\u003c/h2\u003e\\n\")\n                if matched_describe:\n                    f.write(f\"\u003ch3\u003eDescribe\u003c/h3\u003e\u003cpre\u003e{matched_describe}\u003c/pre\u003e\\n\")\n                if matched_logs:\n                    f.write(f\"\u003ch3\u003eLogs\u003c/h3\u003e\u003cpre\u003e{matched_logs}\u003c/pre\u003e\\n\")\n\n            if namespace not in seen_namespaces:\n                events = subprocess.getoutput(f\"kubectl get events -n {namespace}\")\n                matched_events = \"\\n\".join([l for l in events.splitlines() if keyword in l])\n                if matched_events:\n                    namespace_events[namespace] = matched_events\n                seen_namespaces.add(namespace)\n\n        # Write filtered events grouped by namespace\n        for ns, events in namespace_events.items():\n            f.write(f\"\u003ch3\u003eEvents in Namespace: `{ns}`\u003c/h3\u003e\u003cpre\u003e{events}\u003c/pre\u003e\\n\")\n\n        f.write(\"\u003c/body\u003e\u003c/html\u003e\")\n\n    subprocess.run([\"cp\", tmp_report_path, output_report.path], check=True)\n\n"],"image":"python:3.9"}
      ARGO_TEMPLATE:                      {"name":"system-container-driver","inputs":{"parameters":[{"name":"component","value":"{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}"},{"name":"task","value":"{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}"},{"name":"container","value":"{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}"},{"name":"parent-dag-id","value":"125"},{"name":"iteration-index","default":"-1","value":"-1"},{"name":"kubernetes-config","default":"","value":""}]},"outputs":{"parameters":[{"name":"pod-spec-patch","valueFrom":{"path":"/tmp/outputs/pod-spec-patch","default":""}},{"name":"cached-decision","default":"false","valueFrom":{"path":"/tmp/outputs/cached-decision","default":"false"}},{"name":"condition","valueFrom":{"path":"/tmp/outputs/condition","default":"true"}}]},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec","command":["driver"],"args":["--type","CONTAINER","--pipeline_name","pod-log-to-github","--run_id","2e7ffed3-456d-4740-98ac-f89e1c2eb2b9","--dag_execution_id","125","--component","{\"executorLabel\":\"exec-collect-filtered-logs\",\"inputDefinitions\":{\"parameters\":{\"keyword\":{\"defaultValue\":\"crash\",\"isOptional\":true,\"parameterType\":\"STRING\"}}},\"outputDefinitions\":{\"artifacts\":{\"output_report\":{\"artifactType\":{\"schemaTitle\":\"system.Dataset\",\"schemaVersion\":\"0.0.1\"}}}}}","--task","{\"cachingOptions\":{},\"componentRef\":{\"name\":\"comp-collect-filtered-logs\"},\"taskInfo\":{\"name\":\"collect-filtered-logs\"}}","--container","{\"args\":[\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"command\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\"],\"image\":\"python:3.9\"}","--iteration_index","-1","--cached_decision_path","/tmp/outputs/cached-decision","--pod_spec_patch_path","/tmp/outputs/pod-spec-patch","--condition_path","/tmp/outputs/condition","--kubernetes_config",""],"resources":{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-zj76w/2025/05/08/pod-log-to-github-zj76w-system-container-driver-878217725"}}}

Logs (filtered)

time="2025-05-08T05:41:19.919Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-zj76w-system-container-impl-1065118244 (Namespace: kubeflow-user-example-com)

Describe (filtered)

Status:       Failed
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--execution_id\",\"127\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"19\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/2e7ffed3-456d-4740-98ac-f89e1c2eb2b9/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-zj76w/2025/05/08/pod-log-to-github-zj76w-system-container-impl-1065118244"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--execution_id\",\"127\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"19\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/2e7ffed3-456d-4740-98ac-f89e1c2eb2b9/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--execution_id\",\"127\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"19\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/2e7ffed3-456d-4740-98ac-f89e1c2eb2b9/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-zj76w/2025/05/08/pod-log-to-github-zj76w-system-container-impl-1065118244"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--execution_id\",\"127\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"19\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/2e7ffed3-456d-4740-98ac-f89e1c2eb2b9/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--execution_id\",\"127\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"19\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/2e7ffed3-456d-4740-98ac-f89e1c2eb2b9/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-zj76w/2025/05/08/pod-log-to-github-zj76w-system-container-impl-1065118244"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--execution_id\",\"127\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"19\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/2e7ffed3-456d-4740-98ac-f89e1c2eb2b9/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}
              raise FileNotFoundError(f"{report_path} does not exist")
      Reason:       Error
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--execution_id\",\"127\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"19\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/2e7ffed3-456d-4740-98ac-f89e1c2eb2b9/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-zj76w/2025/05/08/pod-log-to-github-zj76w-system-container-impl-1065118244"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--execution_id\",\"127\",\"--executor_input\",\"{\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifacts\\\":[{\\\"name\\\":\\\"19\\\", \\\"type\\\":{\\\"instanceSchema\\\":\\\"\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/2e7ffed3-456d-4740-98ac-f89e1c2eb2b9/collect-filtered-logs/output_report\\\", \\\"metadata\\\":{\\\"display_name\\\":\\\"output_report\\\"}}]}}, \\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\", \\\"github_token\\\":\\\"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\\\", \\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}, \\\"outputs\\\":{\\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}, \\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"main\\\", \\\"isOptional\\\":true}, \\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"}, \\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}, \\\"executorLabel\\\":\\\"exec-upload-to-github\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"'  \\u0026\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef upload_to_github(\\n    report_file: Input[Dataset],\\n    repo_url: str,\\n    github_token: str,\\n    branch: str = \\\"main\\\"\\n):\\n\\n    import subprocess\\n    import shutil\\n    import os\\n    from datetime import datetime\\n\\n    os.environ[\\\"GIT_ASKPASS\\\"] = \\\"echo\\\"\\n\\n    report_path = report_file.path\\n    print(\\\"Checking if report exists at:\\\", report_path)\\n    if not os.path.exists(report_path):\\n        raise FileNotFoundError(f\\\"{report_path} does not exist\\\")\\n\\n    # HTML 파일 이름 만들기\\n    timestamp = datetime.now().strftime(\\\"report-%Y%m%d-%H%M%S.html\\\")\\n    safe_repo_url = repo_url.replace(\\\"https://\\\", f\\\"https://{github_token}@\\\")\\n    repo_dir = \\\"/tmp/repo\\\"\\n    report_target_path = os.path.join(repo_dir, timestamp)\\n\\n    # Git clone\\n    subprocess.run([\\\"git\\\", \\\"clone\\\", \\\"--branch\\\", branch, safe_repo_url, repo_dir], check=True)\\n    shutil.copyfile(report_path, report_target_path)\\n\\n    # Git config\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.email\\\", \\\"ci@example.com\\\"], check=True)\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"config\\\", \\\"user.name\\\", \\\"CI Bot\\\"], check=True)\\n\\n    # Git commit \\u0026 push\\n    subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"add\\\", \\\".\\\"], check=True)\\n    result = subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"status\\\", \\\"--porcelain\\\"], capture_output=True, text=True)\\n\\n    if result.stdout.strip():\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"commit\\\", \\\"-m\\\", f\\\"Add {timestamp}\\\"], check=True)\\n        subprocess.run([\\\"git\\\", \\\"-C\\\", repo_dir, \\\"push\\\"], check=True)\\n        print(f\\\"Report committed and pushed to GitHub as {timestamp}.\\\")\\n    else:\\n        print(\\\"No changes to commit. Skipping.\\\")\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"upload_to_github\"],\"resources\":{}}]}"}

Logs (filtered)

remote: error: GH013: Repository rule violations found for refs/heads/main.        
error: failed to push some refs to 'https://github.com/Rosa1026/Kubeflow.git'
Traceback (most recent call last):
F0508 05:43:13.382344      23 main.go:49] failed to execute component: exit status 1
time="2025-05-08T05:43:14.155Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T05:43:15.067Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-zj76w-system-container-impl-3220014891 (Namespace: kubeflow-user-example-com)

Describe (filtered)

      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--execution_id\",\"126\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/2e7ffed3-456d-4740-98ac-f89e1c2eb2b9/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-zj76w/2025/05/08/pod-log-to-github-zj76w-system-container-impl-3220014891"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--execution_id\",\"126\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/2e7ffed3-456d-4740-98ac-f89e1c2eb2b9/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--execution_id\",\"126\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/2e7ffed3-456d-4740-98ac-f89e1c2eb2b9/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-zj76w/2025/05/08/pod-log-to-github-zj76w-system-container-impl-3220014891"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--execution_id\",\"126\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/2e7ffed3-456d-4740-98ac-f89e1c2eb2b9/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--execution_id\",\"126\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/2e7ffed3-456d-4740-98ac-f89e1c2eb2b9/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-zj76w/2025/05/08/pod-log-to-github-zj76w-system-container-impl-3220014891"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--execution_id\",\"126\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/2e7ffed3-456d-4740-98ac-f89e1c2eb2b9/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}
                  except ValueError:
      ARGO_TEMPLATE:                      {"name":"system-container-impl","inputs":{"parameters":[{"name":"pod-spec-patch","value":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--execution_id\",\"126\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/2e7ffed3-456d-4740-98ac-f89e1c2eb2b9/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}]},"outputs":{},"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}},"container":{"name":"","image":"gcr.io/ml-pipeline/should-be-overridden-during-runtime","command":["should-be-overridden-during-runtime"],"envFrom":[{"configMapRef":{"name":"metadata-grpc-configmap","optional":true}}],"env":[{"name":"KFP_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"KFP_POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}],"resources":{},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]},"volumes":[{"name":"kfp-launcher","emptyDir":{}}],"initContainers":[{"name":"kfp-launcher","image":"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f","command":["launcher-v2","--copy","/kfp-launcher/launch"],"resources":{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m"}},"volumeMounts":[{"name":"kfp-launcher","mountPath":"/kfp-launcher"}]}],"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"minio-service.kubeflow:9000","bucket":"mlpipeline","insecure":true,"accessKeySecret":{"name":"mlpipeline-minio-artifact","key":"accesskey"},"secretKeySecret":{"name":"mlpipeline-minio-artifact","key":"secretkey"},"key":"artifacts/pod-log-to-github-zj76w/2025/05/08/pod-log-to-github-zj76w-system-container-impl-3220014891"}},"podSpecPatch":"{\"containers\":[{\"name\":\"main\",\"image\":\"python:3.9\",\"command\":[\"/var/run/argo/argoexec\",\"emissary\",\"--\",\"/kfp-launcher/launch\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--execution_id\",\"126\",\"--executor_input\",\"{\\\"inputs\\\":{}, \\\"outputs\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifacts\\\":[{\\\"type\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}, \\\"uri\\\":\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/2e7ffed3-456d-4740-98ac-f89e1c2eb2b9/collect-filtered-logs/output_report\\\"}]}}, \\\"outputFile\\\":\\\"/tmp/kfp_outputs/output_metadata.json\\\"}}\",\"--component_spec\",\"{\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"parameterType\\\":\\\"STRING\\\", \\\"defaultValue\\\":\\\"crash\\\", \\\"isOptional\\\":true}}}, \\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\", \\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}, \\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\"}\",\"--pod_name\",\"$(KFP_POD_NAME)\",\"--pod_uid\",\"$(KFP_POD_UID)\",\"--mlmd_server_address\",\"$(METADATA_GRPC_SERVICE_HOST)\",\"--mlmd_server_port\",\"$(METADATA_GRPC_SERVICE_PORT)\",\"--\"],\"args\":[\"sh\",\"-c\",\"\\nif ! [ -x \\\"$(command -v pip)\\\" ]; then\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\nfi\\n\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\u003e=3.7.4,\\u003c5; python_version\\u003c\\\"3.9\\\"' \\u0026\\u0026 \\\"$0\\\" \\\"$@\\\"\\n\",\"sh\",\"-ec\",\"program_path=$(mktemp -d)\\n\\nprintf \\\"%s\\\" \\\"$0\\\" \\u003e \\\"$program_path/ephemeral_component.py\\\"\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\"$program_path/ephemeral_component.py\\\"                         \\\"$@\\\"\\n\",\"\\nimport kfp\\nfrom kfp import dsl\\nfrom kfp.dsl import *\\nfrom typing import *\\n\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\"crash\\\"):\\n    import shutil\\n    import json\\n    import subprocess\\n    import os\\n\\n    tmp_report_path = \\\"/tmp/report.html\\\"\\n    seen_namespaces = set()\\n    namespace_events = {}\\n\\n    # kubectl 설치 방법 변경\\n    subprocess.run([\\\"apt-get\\\", \\\"update\\\", \\\"-y\\\"], check=True)\\n    subprocess.run([ \\n        \\\"curl\\\", \\\"-LO\\\", \\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\"\\n    ], check=True)\\n    subprocess.run([\\\"chmod\\\", \\\"+x\\\", \\\"kubectl\\\"], check=True)\\n    subprocess.run([\\\"mv\\\", \\\"kubectl\\\", \\\"/usr/local/bin/\\\"], check=True)\\n\\n    # Get all pods in all namespaces\\n    pods_output = subprocess.getoutput(\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\")\\n    pod_lines = pods_output.strip().split(\\\"\\\\n\\\")\\n\\n    with open(tmp_report_path, \\\"w\\\") as f:\\n        f.write(f\\\"\\u003chtml\\u003e\\u003cbody\\u003e\\u003ch1\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\u003c/h1\\u003e\\\\n\\\")\\n\\n        for line in pod_lines:\\n            if not line.strip():\\n                continue\\n            try:\\n                namespace, pod_name = line.split()\\n            except ValueError:\\n                continue\\n\\n            describe = subprocess.getoutput(f\\\"kubectl describe pod {pod_name} -n {namespace}\\\")\\n            logs = subprocess.getoutput(f\\\"kubectl logs {pod_name} -n {namespace}\\\")\\n\\n            matched_describe = \\\"\\\\n\\\".join([l for l in describe.splitlines() if keyword in l])\\n            matched_logs = \\\"\\\\n\\\".join([l for l in logs.splitlines() if keyword in l])\\n\\n            if matched_describe or matched_logs:\\n                f.write(f\\\"\\u003ch2\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\u003c/h2\\u003e\\\\n\\\")\\n                if matched_describe:\\n                    f.write(f\\\"\\u003ch3\\u003eDescribe\\u003c/h3\\u003e\\u003cpre\\u003e{matched_describe}\\u003c/pre\\u003e\\\\n\\\")\\n                if matched_logs:\\n                    f.write(f\\\"\\u003ch3\\u003eLogs\\u003c/h3\\u003e\\u003cpre\\u003e{matched_logs}\\u003c/pre\\u003e\\\\n\\\")\\n\\n            if namespace not in seen_namespaces:\\n                events = subprocess.getoutput(f\\\"kubectl get events -n {namespace}\\\")\\n                matched_events = \\\"\\\\n\\\".join([l for l in events.splitlines() if keyword in l])\\n                if matched_events:\\n                    namespace_events[namespace] = matched_events\\n                seen_namespaces.add(namespace)\\n\\n        # Write filtered events grouped by namespace\\n        for ns, events in namespace_events.items():\\n            f.write(f\\\"\\u003ch3\\u003eEvents in Namespace: `{ns}`\\u003c/h3\\u003e\\u003cpre\\u003e{events}\\u003c/pre\\u003e\\\\n\\\")\\n\\n        f.write(\\\"\\u003c/body\\u003e\\u003c/html\\u003e\\\")\\n\\n    subprocess.run([\\\"cp\\\", tmp_report_path, output_report.path], check=True)\\n\\n\",\"--executor_input\",\"{{$}}\",\"--function_to_execute\",\"collect_filtered_logs\"],\"resources\":{}}]}"}

Logs (filtered)

time="2025-05-08T05:42:40.179Z" level=info msg="sub-process exited" argo=true error=""
time="2025-05-08T05:42:41.125Z" level=info msg="sub-process exited" argo=true error=""

Pod: pod-log-to-github-zj76w-system-dag-driver-1372256085 (Namespace: kubeflow-user-example-com)

Logs (filtered)

time="2025-05-08T05:41:09.732Z" level=info msg="sub-process exited" argo=true error=""

Pod: admission-webhook-deployment-6d796b475c-ttrs2 (Namespace: kubeflow)

Logs (filtered)

I0428 06:49:17.823488       1 main.go:771] About to start serving webhooks: &http.Server{Addr:":4443", Handler:http.Handler(nil), DisableGeneralOptionsHandler:false, TLSConfig:(*tls.Config)(0xc0003031e0), ReadTimeout:0, ReadHeaderTimeout:0, WriteTimeout:0, IdleTimeout:0, MaxHeaderBytes:0, TLSNextProto:map[string]func(*http.Server, *tls.Conn, http.Handler)(nil), ConnState:(func(net.Conn, http.ConnState))(nil), ErrorLog:(*log.Logger)(nil), BaseContext:(func(net.Listener) context.Context)(nil), ConnContext:(func(context.Context, net.Conn) context.Context)(nil), inShutdown:atomic.Bool{_:atomic.noCopy{}, v:0x0}, disableKeepAlives:atomic.Bool{_:atomic.noCopy{}, v:0x0}, nextProtoOnce:sync.Once{done:0x0, m:sync.Mutex{state:0, sema:0x0}}, nextProtoErr:error(nil), mu:sync.Mutex{state:0, sema:0x0}, listeners:map[*net.Listener]struct {}(nil), activeConn:map[*http.conn]struct {}(nil), onShutdown:[]func()(nil), listenerGroup:sync.WaitGroup{noCopy:sync.noCopy{}, state:atomic.Uint64{_:atomic.noCopy{}, _:atomic.align64{}, v:0x0}, sema:0x0}}

Pod: katib-controller-6c5684bf6b-l7bcj (Namespace: kubeflow)

Logs (filtered)

{"level":"info","ts":"2025-04-28T06:49:09Z","logger":"trial-controller","msg":"Job watch error. CRD might be missing. Please install CRD and restart katib-controller","CRD Group":"kubeflow.org","CRD Version":"v1","CRD Kind":"TFJob"}
{"level":"info","ts":"2025-04-28T06:49:09Z","logger":"trial-controller","msg":"Job watch error. CRD might be missing. Please install CRD and restart katib-controller","CRD Group":"kubeflow.org","CRD Version":"v1","CRD Kind":"PyTorchJob"}
{"level":"info","ts":"2025-04-28T06:49:09Z","logger":"trial-controller","msg":"Job watch error. CRD might be missing. Please install CRD and restart katib-controller","CRD Group":"kubeflow.org","CRD Version":"v1","CRD Kind":"MPIJob"}
{"level":"info","ts":"2025-04-28T06:49:09Z","logger":"trial-controller","msg":"Job watch error. CRD might be missing. Please install CRD and restart katib-controller","CRD Group":"kubeflow.org","CRD Version":"v1","CRD Kind":"XGBoostJob"}
{"level":"info","ts":"2025-04-28T06:49:09Z","logger":"trial-controller","msg":"Job watch error. CRD might be missing. Please install CRD and restart katib-controller","CRD Group":"kubeflow.org","CRD Version":"v1","CRD Kind":"MXJob"}

Pod: katib-db-manager-7866998cf9-z7lj2 (Namespace: kubeflow)

Logs (filtered)

E0428 06:49:14.661177       1 connection.go:40] Ping to Katib db failed: dial tcp 10.100.199.151:3306: connect: connection refused
E0428 06:49:19.717249       1 connection.go:40] Ping to Katib db failed: dial tcp 10.100.199.151:3306: connect: connection refused
E0428 06:49:24.709335       1 connection.go:40] Ping to Katib db failed: dial tcp 10.100.199.151:3306: connect: connection refused

Pod: metacontroller-0 (Namespace: kubeflow)

Logs (filtered)

{"level":"error","ts":1745823050.650512,"msg":"failed to sync Namespace 'kubeflow-user-example-com': can't reconcile children for Namespace /kubeflow-user-example-com: [Operation cannot be fulfilled on deployments.apps \"ml-pipeline-visualizationserver\": the object has been modified; please apply your changes to the latest version and try again, Operation cannot be fulfilled on deployments.apps \"ml-pipeline-ui-artifact\": the object has been modified; please apply your changes to the latest version and try again]\n","stacktrace":"metacontroller/pkg/controller/composite.(*parentController).worker\n\t/go/src/metacontroller/pkg/controller/composite/controller.go:287\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.21.3/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.21.3/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.21.3/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.21.3/pkg/util/wait/wait.go:90\nmetacontroller/pkg/controller/composite.(*parentController).Start.func1.1\n\t/go/src/metacontroller/pkg/controller/composite/controller.go:263"}
{"level":"error","ts":1745823053.7898242,"msg":"failed to sync Namespace 'kubeflow-user-example-com': can't reconcile children for Namespace /kubeflow-user-example-com: Operation cannot be fulfilled on deployments.apps \"ml-pipeline-ui-artifact\": the object has been modified; please apply your changes to the latest version and try again\n","stacktrace":"metacontroller/pkg/controller/composite.(*parentController).worker\n\t/go/src/metacontroller/pkg/controller/composite/controller.go:287\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.21.3/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.21.3/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.21.3/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.21.3/pkg/util/wait/wait.go:90\nmetacontroller/pkg/controller/composite.(*parentController).Start.func1.1\n\t/go/src/metacontroller/pkg/controller/composite/controller.go:263"}

Pod: metadata-envoy-deployment-9c7db86d8-h96xq (Namespace: kubeflow)

Logs (filtered)

[2025-04-28 06:44:47.431][1][info][main] [source/server/server.cc:473]   response header map: 440 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-allow-private-network,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,proxy-status,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id

Pod: metadata-grpc-deployment-d94cc8676-njh69 (Namespace: kubeflow)

Describe (filtered)

      Reason:       Error

Logs (filtered)

W0429 01:07:06.953629    11 metadata_store_service_impl.cc:239] GetContextType failed: No type found for query, name: `system.Pipeline`, version: `nullopt`
W0429 01:07:07.075496    11 metadata_store_service_impl.cc:239] GetContextType failed: No type found for query, name: `system.PipelineRun`, version: `nullopt`

Pod: metadata-writer-d9bc4bb89-bd6nz (Namespace: kubeflow)

Logs (filtered)

Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):

Pod: ml-pipeline-5846b5b56d-l8d7n (Namespace: kubeflow)

Logs (filtered)

E0428 07:09:33.152184       7 error.go:280] Unauthorized access
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:227
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to list runs due to authorization error. Check if you have permission to access namespace undefined
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
/api.RunService/ListRunsV1 call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0428 07:12:32.305055       7 error.go:278] Invalid input error: An experiment cannot have an empty namespace in multi-user mode
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:185
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
List experiments failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
/kubeflow.pipelines.backend.api.v2beta1.ExperimentService/ListExperiments call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0429 01:02:24.849769       7 error.go:278] Invalid input error: Experiment id can not be empty in multi-user mode
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:185
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
/kubeflow.pipelines.backend.api.v2beta1.RunService/CreateRun call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0430 01:46:31.802286       7 error.go:278] workflows.argoproj.io "review-pipeline-slndx" not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:162
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
/api.ReportService/ReportWorkflowV1 call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
E0507 01:40:55.007968       7 error.go:280] Unauthorized access
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:227
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to list runs due to authorization error. Check if you have permission to access namespace undefined
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
/api.RunService/ListRunsV1 call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 01:41:22.958737       7 error.go:278] Invalid input error: A run cannot have an empty namespace in multi-user mode
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:185
Failed to list runs due to authorization error. Check if you have permission to access namespace 
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
/kubeflow.pipelines.backend.api.v2beta1.RunService/ListRuns call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
E0507 02:45:44.346277       7 resource_manager.go:1270] pipeline 'review-pipeline-ph7jk' finished with an error
E0507 02:50:44.894156       7 resource_manager.go:1270] pipeline 'review-pipeline-9wl4l' finished with an error
E0507 04:27:06.878961       7 resource_manager.go:1270] pipeline 'review-pipeline-2hczt' finished with an error
I0507 04:30:10.439664       7 error.go:278] Invalid input error: Failed to delete pipeline with id 5aaa132c-64a5-4785-9e06-3d94e2b4e9c9 as it has existing pipeline versions (e.g. dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c)
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:185
Failed to delete pipeline 5aaa132c-64a5-4785-9e06-3d94e2b4e9c9. Check error stack
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/DeletePipeline call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:30:10.440324       7 error.go:278] Invalid input error: Failed to delete pipeline with id a975c325-7ec3-4f97-be35-0e8efb3194b7 as it has existing pipeline versions (e.g. a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c)
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:185
Failed to delete pipeline a975c325-7ec3-4f97-be35-0e8efb3194b7. Check error stack
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/DeletePipeline call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:30:10.445552       7 error.go:278] Invalid input error: Failed to delete pipeline with id 56eb0cec-4454-49f5-8ac8-038b5ce47560 as it has existing pipeline versions (e.g. f0d68c4b-809a-4be6-9557-12649e951e01)
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:185
Failed to delete pipeline 56eb0cec-4454-49f5-8ac8-038b5ce47560. Check error stack
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/DeletePipeline call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:30:30.661217       7 error.go:278] ResourceNotFoundError: PipelineVersion 5aaa132c-64a5-4785-9e06-3d94e2b4e9c9 not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
Failed to get a pipeline (v1beta1) 5aaa132c-64a5-4785-9e06-3d94e2b4e9c9 due to error fetching the latest pipeline version
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/api.PipelineService/GetPipelineV1 call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:30:33.945489       7 error.go:278] ResourceNotFoundError: PipelineVersion 5aaa132c-64a5-4785-9e06-3d94e2b4e9c9 not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
Failed to get a pipeline (v1beta1) 5aaa132c-64a5-4785-9e06-3d94e2b4e9c9 due to error fetching the latest pipeline version
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/api.PipelineService/GetPipelineV1 call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:30:35.730119       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:30:35.746894       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:30:41.963188       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:30:41.981154       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:30:44.692280       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:30:44.709377       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:30:56.497429       7 error.go:278] ResourceNotFoundError: PipelineVersion 5aaa132c-64a5-4785-9e06-3d94e2b4e9c9 not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
Failed to get a pipeline (v1beta1) 5aaa132c-64a5-4785-9e06-3d94e2b4e9c9 due to error fetching the latest pipeline version
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/api.PipelineService/GetPipelineV1 call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:30:59.509329       7 error.go:278] ResourceNotFoundError: PipelineVersion 5aaa132c-64a5-4785-9e06-3d94e2b4e9c9 not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
Failed to get a pipeline (v1beta1) 5aaa132c-64a5-4785-9e06-3d94e2b4e9c9 due to error fetching the latest pipeline version
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/api.PipelineService/GetPipelineV1 call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:31:10.812235       7 error.go:278] Invalid input error: Failed to delete pipeline with id a975c325-7ec3-4f97-be35-0e8efb3194b7 as it has existing pipeline versions (e.g. a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c)
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:185
Failed to delete pipeline a975c325-7ec3-4f97-be35-0e8efb3194b7. Check error stack
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/DeletePipeline call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:31:13.515726       7 error.go:278] Invalid input error: Failed to delete pipeline with id a975c325-7ec3-4f97-be35-0e8efb3194b7 as it has existing pipeline versions (e.g. a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c)
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:185
Failed to delete pipeline a975c325-7ec3-4f97-be35-0e8efb3194b7. Check error stack
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/DeletePipeline call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:31:17.697476       7 error.go:278] Invalid input error: Failed to delete pipeline with id a975c325-7ec3-4f97-be35-0e8efb3194b7 as it has existing pipeline versions (e.g. a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c)
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:185
Failed to delete pipeline a975c325-7ec3-4f97-be35-0e8efb3194b7. Check error stack
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/DeletePipeline call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:31:28.175426       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:31:28.175442       7 error.go:278] ResourceNotFoundError: PipelineVersion a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:31:28.196027       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:31:33.458500       7 error.go:278] ResourceNotFoundError: PipelineVersion a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:31:33.467576       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:31:33.485180       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:31:41.912369       7 error.go:278] ResourceNotFoundError: PipelineVersion a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:31:41.913421       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:31:41.932012       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:31:51.392606       7 error.go:278] ResourceNotFoundError: PipelineVersion a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:31:51.400529       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:31:51.416611       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:32:00.519844       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:32:00.527641       7 error.go:278] ResourceNotFoundError: PipelineVersion a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:32:00.536180       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:32:03.432279       7 error.go:278] ResourceNotFoundError: PipelineVersion a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:32:03.432604       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:32:03.449193       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:32:15.250787       7 error.go:278] ResourceNotFoundError: PipelineVersion a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:32:15.253861       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:32:15.269353       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:32:37.571768       7 error.go:278] ResourceNotFoundError: PipelineVersion a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:32:37.581855       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 04:32:37.601263       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
E0507 04:34:18.649119       7 resource_manager.go:1270] pipeline 'review-pipeline-w64f8' finished with an error
E0507 04:49:17.281672       7 resource_manager.go:1270] pipeline 'review-pipeline-m8w8c' finished with an error
E0507 05:12:57.789443       7 resource_manager.go:1270] pipeline 'pod-log-to-github-2s5r9' finished with an error
I0507 05:23:19.527691       7 error.go:278] Error 3988 (HY000): Conversion from collation utf8_general_ci into utf8mb4_0900_ai_ci impossible for parameter
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:144
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
/kubeflow.pipelines.backend.api.v2beta1.RunService/CreateRun call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 05:23:20.543224       7 error.go:278] ResourceNotFoundError: Run 00486f47-308e-4d46-8b38-4dea3117e3bb not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
/api.ReportService/ReportWorkflowV1 call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 05:23:37.304364       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 05:23:37.307535       7 error.go:278] ResourceNotFoundError: PipelineVersion a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 05:23:37.322487       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 05:23:38.610771       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 05:23:38.620269       7 error.go:278] ResourceNotFoundError: PipelineVersion a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 05:23:38.627938       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 05:24:09.071371       7 error.go:278] Error 3988 (HY000): Conversion from collation utf8_general_ci into utf8mb4_0900_ai_ci impossible for parameter
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:144
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
/kubeflow.pipelines.backend.api.v2beta1.RunService/CreateRun call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 05:31:39.939404       7 error.go:278] ResourceNotFoundError: PipelineVersion a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 05:31:39.948542       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 05:31:39.962703       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 05:31:56.892834       7 error.go:278] Invalid input error: Failed to delete pipeline with id 56eb0cec-4454-49f5-8ac8-038b5ce47560 as it has existing pipeline versions (e.g. ea622cb5-1cb9-4d70-bd58-30d81524282b)
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:185
Failed to delete pipeline 56eb0cec-4454-49f5-8ac8-038b5ce47560. Check error stack
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/DeletePipeline call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 05:31:59.938475       7 error.go:278] Invalid input error: Failed to delete pipeline with id 56eb0cec-4454-49f5-8ac8-038b5ce47560 as it has existing pipeline versions (e.g. ea622cb5-1cb9-4d70-bd58-30d81524282b)
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:185
Failed to delete pipeline 56eb0cec-4454-49f5-8ac8-038b5ce47560. Check error stack
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/DeletePipeline call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 05:32:01.960332       7 error.go:278] ResourceNotFoundError: PipelineVersion a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 05:32:01.998254       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 05:32:02.018687       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
E0507 05:48:49.622259       7 resource_manager.go:1270] pipeline 'pod-log-to-github-2tkgb' finished with an error
I0507 05:49:26.148203       7 error.go:278] ResourceNotFoundError: Run 36bc2918-ed57-4444-847e-fd6d3971347d not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
/api.ReportService/ReportWorkflowV1 call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
E0507 05:49:32.256689       7 resource_manager.go:1270] pipeline 'pod-log-to-github-gfhgz' finished with an error
E0507 05:50:06.068458       7 resource_manager.go:1270] pipeline 'pod-log-to-github-xkz8p' finished with an error
E0507 05:50:09.204478       7 resource_manager.go:1270] pipeline 'pod-log-to-github-t4w5w' finished with an error
E0507 05:50:38.423751       7 resource_manager.go:1270] pipeline 'pod-log-to-github-qv4f5' finished with an error
I0507 05:54:25.136740       7 error.go:278] ResourceNotFoundError: PipelineVersion a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 05:54:25.137350       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 05:56:24.240610       7 error.go:278] ResourceNotFoundError: PipelineVersion a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 05:56:24.244028       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
E0507 05:56:43.029846       7 resource_manager.go:1270] pipeline 'pod-log-to-github-gfhgz' finished with an error
I0507 05:57:08.114551       7 error.go:278] Invalid input error: A run cannot have an empty namespace in multi-user mode
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:185
Failed to list runs due to authorization error. Check if you have permission to access namespace 
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
/kubeflow.pipelines.backend.api.v2beta1.RunService/ListRuns call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 05:57:08.407493       7 error.go:278] ResourceNotFoundError: PipelineVersion a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 05:57:08.423652       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 05:57:09.640292       7 error.go:278] ResourceNotFoundError: PipelineVersion a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 05:57:09.667700       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 06:24:16.295527       7 error.go:278] ResourceNotFoundError: PipelineVersion a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 06:24:16.305792       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 06:24:21.679778       7 error.go:278] ResourceNotFoundError: PipelineVersion a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 06:24:21.685507       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
E0507 06:54:44.825573       7 resource_manager.go:1270] pipeline 'pod-log-to-github-7xk8h' finished with an error
E0507 07:46:50.036876       7 resource_manager.go:1270] pipeline 'pod-log-to-github-6lxcf' finished with an error
E0507 07:47:57.066598       7 resource_manager.go:1270] pipeline 'pod-log-to-github-qlsrg' finished with an error
E0507 07:48:39.766053       7 resource_manager.go:1270] pipeline 'pod-log-to-github-vpkhs' finished with an error
E0507 07:50:25.419322       7 resource_manager.go:1270] pipeline 'pod-log-to-github-2lp4x' finished with an error
E0507 07:50:37.431773       7 resource_manager.go:1270] pipeline 'pod-log-to-github-cdd4s' finished with an error
I0507 07:52:51.659095       7 error.go:278] ResourceNotFoundError: PipelineVersion a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 07:52:51.674135       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0507 07:52:51.699642       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
E0507 07:52:57.424778       7 resource_manager.go:1270] pipeline 'pod-log-to-github-z9lb2' finished with an error
E0507 07:57:05.040167       7 resource_manager.go:1270] pipeline 'pod-log-to-github-vc5rx' finished with an error
E0507 08:04:57.721555       7 resource_manager.go:1270] pipeline 'pod-log-to-github-dnplr' finished with an error
I0508 00:24:51.440178       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0508 00:24:51.467011       7 error.go:278] ResourceNotFoundError: PipelineVersion a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id a9f50d1b-fff6-46d3-a367-f6e33e3f1f5c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0508 00:24:52.442812       7 error.go:278] ResourceNotFoundError: PipelineVersion dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
Failed to get a pipeline version due to authorization error for pipeline version id dc01c2c4-ad8b-4b96-a565-1e4aab55aa6c
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
/kubeflow.pipelines.backend.api.v2beta1.PipelineService/GetPipelineVersion call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
E0508 00:46:26.761508       7 resource_manager.go:1270] pipeline 'pod-log-to-github-nqs86' finished with an error
E0508 00:46:42.685039       7 resource_manager.go:1270] pipeline 'pod-log-to-github-sqv6q' finished with an error
E0508 00:47:37.999830       7 resource_manager.go:1270] pipeline 'pod-log-to-github-lq7m2' finished with an error
E0508 00:50:12.475816       7 resource_manager.go:1270] pipeline 'pod-log-to-github-vkpgx' finished with an error
E0508 01:03:33.731772       7 resource_manager.go:1270] pipeline 'pod-log-to-github-nqs86' finished with an error
E0508 01:05:57.349069       7 resource_manager.go:1270] pipeline 'pod-log-to-github-kcp9j' finished with an error
E0508 01:11:44.735502       7 resource_manager.go:1270] pipeline 'pod-log-to-github-lm5jl' finished with an error
E0508 01:52:05.195831       7 resource_manager.go:1270] pipeline 'pod-log-to-github-5sf2x' finished with an error
E0508 01:56:47.434680       7 resource_manager.go:1270] pipeline 'pod-log-to-github-5sf2x' finished with an error
I0508 02:44:57.502819       7 error.go:278] workflows.argoproj.io "review-pipeline-9wl4l" not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:162
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
/api.ReportService/ReportWorkflowV1 call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0508 02:46:27.513899       7 error.go:278] workflows.argoproj.io "review-pipeline-ph7jk" not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:162
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
/api.ReportService/ReportWorkflowV1 call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0508 04:44:57.690446       7 error.go:278] workflows.argoproj.io "review-pipeline-m8w8c" not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:162
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
/api.ReportService/ReportWorkflowV1 call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
E0508 04:45:24.137306       7 pipeline_upload_server.go:299] Failed to upload pipelines. Error: Failed to create a pipeline and a pipeline version. The pipeline already exists.: Failed to create a pipeline and a pipeline version: Already exist error: Failed to create a new pipeline. The name pod_log_pipeline already exists. Please specify a new name
E0508 04:46:56.774930       7 resource_manager.go:1270] pipeline 'pod-log-to-github-85dff' finished with an error
I0508 04:46:57.699285       7 error.go:278] workflows.argoproj.io "review-pipeline-2hczt" not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:162
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
/api.ReportService/ReportWorkflowV1 call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0508 04:47:57.690354       7 error.go:278] workflows.argoproj.io "review-pipeline-w64f8" not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:162
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
/api.ReportService/ReportWorkflowV1 call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
E0508 04:53:55.181041       7 resource_manager.go:1270] pipeline 'pod-log-to-github-phkw4' finished with an error
E0508 04:57:07.333619       7 resource_manager.go:1270] pipeline 'pod-log-to-github-mml78' finished with an error
E0508 05:05:18.502550       7 resource_manager.go:1270] pipeline 'pod-log-to-github-zctnf' finished with an error
I0508 05:05:21.413294       7 error.go:278] Invalid input error: Experiment id can not be empty in multi-user mode
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:185
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
/kubeflow.pipelines.backend.api.v2beta1.RunService/CreateRun call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
E0508 05:07:49.217797       7 resource_manager.go:1270] pipeline 'pod-log-to-github-ljqp8' finished with an error
I0508 05:13:23.684552       7 error.go:278] workflows.argoproj.io "pod-log-to-github-2s5r9" not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:162
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
/api.ReportService/ReportWorkflowV1 call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0508 05:26:23.696434       7 error.go:278] workflows.argoproj.io "pod-log-to-github-t4w5w" not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:162
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
/api.ReportService/ReportWorkflowV1 call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
E0508 05:28:12.264384       7 resource_manager.go:1270] pipeline 'pod-log-to-github-8mjdq' finished with an error
I0508 05:32:23.709058       7 error.go:278] workflows.argoproj.io "pod-log-to-github-xkz8p" not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:162
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
/api.ReportService/ReportWorkflowV1 call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0508 05:37:23.720666       7 error.go:278] workflows.argoproj.io "pod-log-to-github-2tkgb" not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:162
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
/api.ReportService/ReportWorkflowV1 call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0508 05:44:53.724112       7 error.go:278] workflows.argoproj.io "pod-log-to-github-qv4f5" not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:162
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
/api.ReportService/ReportWorkflowV1 call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
E0508 05:47:51.504577       7 resource_manager.go:1270] pipeline 'pod-log-to-github-zj76w' finished with an error
E0508 05:53:11.431594       7 resource_manager.go:1270] pipeline 'pod-log-to-github-scgtw' finished with an error
E0508 05:56:48.851424       7 resource_manager.go:1270] pipeline 'pod-log-to-github-pm5rh' finished with an error
I0508 05:56:53.741426       7 error.go:278] workflows.argoproj.io "pod-log-to-github-gfhgz" not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:162
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
/api.ReportService/ReportWorkflowV1 call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
E0508 06:00:12.285360       7 resource_manager.go:1270] pipeline 'pod-log-to-github-tvwg6' finished with an error
E0508 06:04:17.774140       7 resource_manager.go:1270] pipeline 'pod-log-to-github-tvwg6' finished with an error
E0508 06:15:05.656244       7 resource_manager.go:1270] pipeline 'pod-log-to-github-nm76f' finished with an error
I0508 06:28:52.360719       7 error.go:278] Error 3988 (HY000): Conversion from collation utf8_general_ci into utf8mb4_0900_ai_ci impossible for parameter
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:144
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
/kubeflow.pipelines.backend.api.v2beta1.RunService/CreateRun call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0508 06:28:53.376739       7 error.go:278] ResourceNotFoundError: Run ecfd93e6-166e-4811-920a-fb70c4edd9f1 not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
/api.ReportService/ReportWorkflowV1 call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0508 06:28:57.633115       7 error.go:278] Error 3988 (HY000): Conversion from collation utf8_general_ci into utf8mb4_0900_ai_ci impossible for parameter
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:144
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
/kubeflow.pipelines.backend.api.v2beta1.RunService/CreateRun call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
I0508 06:28:58.648532       7 error.go:278] ResourceNotFoundError: Run d1b8ce1d-db0c-401a-9d6a-4e8b08c51496 not found
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:170
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:271
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:350
/api.ReportService/ReportWorkflowV1 call failed
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:266
	/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:337

Pod: ml-pipeline-persistenceagent-7655ddbcfb-mnjmg (Namespace: kubeflow)

Logs (filtered)

time="2025-05-08T05:37:23Z" level=error msg="Permanent failure while syncing resource (kubeflow-user-example-com/pod-log-to-github-2tkgb): CustomError (code: 1): Syncing Workflow (pod-log-to-github-2tkgb): permanent failure: CustomError (code: 1): Error while reporting workflow resource (code: NotFound, message: Failed to report workflow: NotFoundError: Failed to add PersistedFinalState label to workflow pod-log-to-github-2tkgb: workflows.argoproj.io \"pod-log-to-github-2tkgb\" not found): rpc error: code = NotFound desc = Failed to report workflow: NotFoundError: Failed to add PersistedFinalState label to workflow pod-log-to-github-2tkgb: workflows.argoproj.io \"pod-log-to-github-2tkgb\" not found, {\"metadata\":{\"name\":\"pod-log-to-github-2tkgb\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"a5fd8c28-c792-45f3-9853-a321205997e9\",\"resourceVersion\":\"6048612\",\"generation\":9,\"creationTimestamp\":\"2025-05-07T05:36:04Z\",\"labels\":{\"pipeline/persistedFinalState\":\"true\",\"pipeline/runid\":\"f994665a-c6dc-432d-a15f-07517f9c4f4d\",\"workflows.argoproj.io/completed\":\"true\",\"workflows.argoproj.io/phase\":\"Failed\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-pod-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-pod-logs\\\",\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"pod_name\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}},\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-pod-logs\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-pod-logs\\\"},\\\"inputs\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"componentInputParameter\\\":\\\"namespace\\\"},\\\"pod_name\\\":{\\\"componentInputParameter\\\":\\\"pod_name\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-pod-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-pod-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-pod-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"namespace\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"pod_name\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-pod-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_pod_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_pod_logs(\\\\n    namespace: str,\\\\n    pod_name: str,\\\\n    output_report: Output[Dataset]\\\\n):\\\\n    import subprocess\\\\n\\\\n    describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n    logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n    events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n\\\\n    with open(output_report.path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(\\\\\\\"# 장애 복기 리포트\\\\\\\\n\\\\\\\")\\\\n        f.write(f\\\\\\\"**Pod**: {pod_name}\\\\\\\\n\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## describe\\\\\\\\n```text\\\\\\\\n\\\\\\\" + describe + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## logs\\\\\\\\n```text\\\\\\\\n\\\\\\\" + logs + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## events\\\\\\\\n```text\\\\\\\\n\\\\\\\" + events + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str = \\\\\\\"main\\\\\\\"\\\\n):\\\\n    import os\\\\n    import subprocess\\\\n\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_file.path)\\\\n    if not os.path.exists(report_file.path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_file.path} does not exist\\\\\\\")\\\\n\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", report_file.path, os.path.join(repo_dir, \\\\\\\"report.md\\\\\\\")], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", \\\\\\\"Add report\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Run of pod_log_pipeline_version_at_2025-05-07T05:35:37.141Z (40a7e)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-07T05:37:15Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/completed\":{},\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}},{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-07T05:48:49Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-pod-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-pod-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/persistedFinalState\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"f994665a-c6dc-432d-a15f-07517f9c4f4d\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-pod-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-pod-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-pod-logs\\\"},\\\"inputs\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"componentInputParameter\\\":\\\"namespace\\\"},\\\"pod_name\\\":{\\\"componentInputParameter\\\":\\\"pod_name\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-pod-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-pod-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-pod-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-pod-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-pod-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-pod-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-pod-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-pod-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-pod-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"f994665a-c6dc-432d-a15f-07517f9c4f4d\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\",\\\"github_token\\\":\\\"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\\\",\\\"namespace\\\":\\\"istio-system\\\",\\\"pod_name\\\":\\\"toolbox\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"f994665a-c6dc-432d-a15f-07517f9c4f4d\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Failed\",\"startedAt\":\"2025-05-07T05:36:04Z\",\"finishedAt\":\"2025-05-07T05:37:15Z\",\"progress\":\"4/5\",\"nodes\":{\"pod-log-to-github-2tkgb\":{\"id\":\"pod-log-to-github-2tkgb\",\"name\":\"pod-log-to-github-2tkgb\",\"displayName\":\"pod-log-to-github-2tkgb\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-2tkgb\",\"phase\":\"Failed\",\"startedAt\":\"2025-05-07T05:36:04Z\",\"finishedAt\":\"2025-05-07T05:37:15Z\",\"progress\":\"4/5\",\"resourcesDuration\":{\"cpu\":41,\"memory\":38},\"children\":[\"pod-log-to-github-2tkgb-830918901\"],\"outboundNodes\":[\"pod-log-to-github-2tkgb-819368644\"]},\"pod-log-to-github-2tkgb-1972551914\":{\"id\":\"pod-log-to-github-2tkgb-1972551914\",\"name\":\"pod-log-to-github-2tkgb.root.upload-to-github-driver\",\"displayName\":\"upload-to-github-driver\",\"type\":\"Pod\",\"templateName\":\"system-container-driver\",\"templateScope\":\"local/pod-log-to-github-2tkgb\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-2tkgb-2441106536\",\"startedAt\":\"2025-05-07T05:36:45Z\",\"finishedAt\":\"2025-05-07T05:36:48Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":4,\"memory\":3},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-pod-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-pod-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str = \\\\\\\"main\\\\\\\"\\\\n):\\\\n    import os\\\\n    import subprocess\\\\n\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_file.path)\\\\n    if not os.path.exists(report_file.path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_file.path} does not exist\\\\\\\")\\\\n\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", report_file.path, os.path.join(repo_dir, \\\\\\\"report.md\\\\\\\")], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", \\\\\\\"Add report\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\"},{\"name\":\"parent-dag-id\",\"value\":\"30\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\",\"value\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"python:3.9\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"f994665a-c6dc-432d-a15f-07517f9c4f4d\\\",\\\"--execution_id\\\",\\\"32\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"report_file\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"name\\\\\\\":\\\\\\\"1\\\\\\\", \\\\\\\"type\\\\\\\":{\\\\\\\"instanceSchema\\\\\\\":\\\\\\\"\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/f994665a-c6dc-432d-a15f-07517f9c4f4d/collect-pod-logs/output_report\\\\\\\", \\\\\\\"metadata\\\\\\\":{\\\\\\\"display_name\\\\\\\":\\\\\\\"output_report\\\\\\\"}}]}}, \\\\\\\"parameterValues\\\\\\\":{\\\\\\\"branch\\\\\\\":\\\\\\\"main\\\\\\\", \\\\\\\"github_token\\\\\\\":\\\\\\\"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\\\\\\\", \\\\\\\"repo_url\\\\\\\":\\\\\\\"https://github.com/Rosa1026/Kubeflow.git\\\\\\\"}}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"inputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"report_file\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}, \\\\\\\"parameters\\\\\\\":{\\\\\\\"branch\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\", \\\\\\\"defaultValue\\\\\\\":\\\\\\\"main\\\\\\\", \\\\\\\"isOptional\\\\\\\":true}, \\\\\\\"github_token\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}, \\\\\\\"repo_url\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-upload-to-github\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str = \\\\\\\"main\\\\\\\"\\\\n):\\\\n    import os\\\\n    import subprocess\\\\n\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_file.path)\\\\n    if not os.path.exists(report_file.path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_file.path} does not exist\\\\\\\")\\\\n\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", report_file.path, os.path.join(repo_dir, \\\\\\\"report.md\\\\\\\")], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", \\\\\\\"Add report\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"resources\\\":{}}]}\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-2tkgb/2025/05/07/pod-log-to-github-2tkgb-system-container-driver-1972551914/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-2tkgb-969796849\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-2tkgb-2441106536\":{\"id\":\"pod-log-to-github-2tkgb-2441106536\",\"name\":\"pod-log-to-github-2tkgb.root\",\"displayName\":\"root\",\"type\":\"DAG\",\"templateName\":\"root\",\"templateScope\":\"local/pod-log-to-github-2tkgb\",\"phase\":\"Failed\",\"boundaryID\":\"pod-log-to-github-2tkgb\",\"startedAt\":\"2025-05-07T05:36:14Z\",\"finishedAt\":\"2025-05-07T05:37:15Z\",\"progress\":\"3/4\",\"resourcesDuration\":{\"cpu\":38,\"memory\":36},\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"30\"}]},\"children\":[\"pod-log-to-github-2tkgb-3003919539\"],\"outboundNodes\":[\"pod-log-to-github-2tkgb-819368644\"]},\"pod-log-to-github-2tkgb-3003919539\":{\"id\":\"pod-log-to-github-2tkgb-3003919539\",\"name\":\"pod-log-to-github-2tkgb.root.collect-pod-logs-driver\",\"displayName\":\"collect-pod-logs-driver\",\"type\":\"Pod\",\"templateName\":\"system-container-driver\",\"templateScope\":\"local/pod-log-to-github-2tkgb\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-2tkgb-2441106536\",\"startedAt\":\"2025-05-07T05:36:14Z\",\"finishedAt\":\"2025-05-07T05:36:18Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":3,\"memory\":2},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"executorLabel\\\":\\\"exec-collect-pod-logs\\\",\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"pod_name\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}},\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-pod-logs\\\"},\\\"inputs\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"componentInputParameter\\\":\\\"namespace\\\"},\\\"pod_name\\\":{\\\"componentInputParameter\\\":\\\"pod_name\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-pod-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_pod_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_pod_logs(\\\\n    namespace: str,\\\\n    pod_name: str,\\\\n    output_report: Output[Dataset]\\\\n):\\\\n    import subprocess\\\\n\\\\n    describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n    logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n    events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n\\\\n    with open(output_report.path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(\\\\\\\"# 장애 복기 리포트\\\\\\\\n\\\\\\\")\\\\n        f.write(f\\\\\\\"**Pod**: {pod_name}\\\\\\\\n\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## describe\\\\\\\\n```text\\\\\\\\n\\\\\\\" + describe + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## logs\\\\\\\\n```text\\\\\\\\n\\\\\\\" + logs + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## events\\\\\\\\n```text\\\\\\\\n\\\\\\\" + events + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\"},{\"name\":\"parent-dag-id\",\"value\":\"30\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\",\"value\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"python:3.9\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"f994665a-c6dc-432d-a15f-07517f9c4f4d\\\",\\\"--execution_id\\\",\\\"31\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{\\\\\\\"parameterValues\\\\\\\":{\\\\\\\"namespace\\\\\\\":\\\\\\\"istio-system\\\\\\\", \\\\\\\"pod_name\\\\\\\":\\\\\\\"toolbox\\\\\\\"}}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/f994665a-c6dc-432d-a15f-07517f9c4f4d/collect-pod-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"inputDefinitions\\\\\\\":{\\\\\\\"parameters\\\\\\\":{\\\\\\\"namespace\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}, \\\\\\\"pod_name\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}}}, \\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-pod-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_pod_logs(\\\\n    namespace: str,\\\\n    pod_name: str,\\\\n    output_report: Output[Dataset]\\\\n):\\\\n    import subprocess\\\\n\\\\n    describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n    logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n    events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n\\\\n    with open(output_report.path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(\\\\\\\"# 장애 복기 리포트\\\\\\\\n\\\\\\\")\\\\n        f.write(f\\\\\\\"**Pod**: {pod_name}\\\\\\\\n\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## describe\\\\\\\\n```text\\\\\\\\n\\\\\\\" + describe + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## logs\\\\\\\\n```text\\\\\\\\n\\\\\\\" + logs + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## events\\\\\\\\n```text\\\\\\\\n\\\\\\\" + events + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_pod_logs\\\"],\\\"resources\\\":{}}]}\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-2tkgb/2025/05/07/pod-log-to-github-2tkgb-system-container-driver-3003919539/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-2tkgb-3928305142\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-2tkgb-3928305142\":{\"id\":\"pod-log-to-github-2tkgb-3928305142\",\"name\":\"pod-log-to-github-2tkgb.root.collect-pod-logs\",\"displayName\":\"collect-pod-logs\",\"type\":\"DAG\",\"templateName\":\"system-container-executor\",\"templateScope\":\"local/pod-log-to-github-2tkgb\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-2tkgb-2441106536\",\"startedAt\":\"2025-05-07T05:36:24Z\",\"finishedAt\":\"2025-05-07T05:36:45Z\",\"progress\":\"2/3\",\"resourcesDuration\":{\"cpu\":35,\"memory\":34},\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"python:3.9\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"f994665a-c6dc-432d-a15f-07517f9c4f4d\\\",\\\"--execution_id\\\",\\\"31\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{\\\\\\\"parameterValues\\\\\\\":{\\\\\\\"namespace\\\\\\\":\\\\\\\"istio-system\\\\\\\", \\\\\\\"pod_name\\\\\\\":\\\\\\\"toolbox\\\\\\\"}}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/f994665a-c6dc-432d-a15f-07517f9c4f4d/collect-pod-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"inputDefinitions\\\\\\\":{\\\\\\\"parameters\\\\\\\":{\\\\\\\"namespace\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}, \\\\\\\"pod_name\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}}}, \\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-pod-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_pod_logs(\\\\n    namespace: str,\\\\n    pod_name: str,\\\\n    output_report: Output[Dataset]\\\\n):\\\\n    import subprocess\\\\n\\\\n    describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n    logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n    events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n\\\\n    with open(output_report.path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(\\\\\\\"# 장애 복기 리포트\\\\\\\\n\\\\\\\")\\\\n        f.write(f\\\\\\\"**Pod**: {pod_name}\\\\\\\\n\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## describe\\\\\\\\n```text\\\\\\\\n\\\\\\\" + describe + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## logs\\\\\\\\n```text\\\\\\\\n\\\\\\\" + logs + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## events\\\\\\\\n```text\\\\\\\\n\\\\\\\" + events + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_pod_logs\\\"],\\\"resources\\\":{}}]}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"false\"}]},\"children\":[\"pod-log-to-github-2tkgb-610323093\"],\"outboundNodes\":[\"pod-log-to-github-2tkgb-610323093\"]},\"pod-log-to-github-2tkgb-610323093\":{\"id\":\"pod-log-to-github-2tkgb-610323093\",\"name\":\"pod-log-to-github-2tkgb.root.collect-pod-logs.executor\",\"displayName\":\"executor\",\"type\":\"Pod\",\"templateName\":\"system-container-impl\",\"templateScope\":\"local/pod-log-to-github-2tkgb\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-2tkgb-3928305142\",\"startedAt\":\"2025-05-07T05:36:24Z\",\"finishedAt\":\"2025-05-07T05:36:34Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":14,\"memory\":14},\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"python:3.9\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"f994665a-c6dc-432d-a15f-07517f9c4f4d\\\",\\\"--execution_id\\\",\\\"31\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{\\\\\\\"parameterValues\\\\\\\":{\\\\\\\"namespace\\\\\\\":\\\\\\\"istio-system\\\\\\\", \\\\\\\"pod_name\\\\\\\":\\\\\\\"toolbox\\\\\\\"}}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/f994665a-c6dc-432d-a15f-07517f9c4f4d/collect-pod-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"inputDefinitions\\\\\\\":{\\\\\\\"parameters\\\\\\\":{\\\\\\\"namespace\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}, \\\\\\\"pod_name\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}}}, \\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-pod-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_pod_logs(\\\\n    namespace: str,\\\\n    pod_name: str,\\\\n    output_report: Output[Dataset]\\\\n):\\\\n    import subprocess\\\\n\\\\n    describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n    logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n    events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n\\\\n    with open(output_report.path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(\\\\\\\"# 장애 복기 리포트\\\\\\\\n\\\\\\\")\\\\n        f.write(f\\\\\\\"**Pod**: {pod_name}\\\\\\\\n\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## describe\\\\\\\\n```text\\\\\\\\n\\\\\\\" + describe + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## logs\\\\\\\\n```text\\\\\\\\n\\\\\\\" + logs + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## events\\\\\\\\n```text\\\\\\\\n\\\\\\\" + events + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_pod_logs\\\"],\\\"resources\\\":{}}]}\"}]},\"outputs\":{\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-2tkgb/2025/05/07/pod-log-to-github-2tkgb-system-container-impl-610323093/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-2tkgb-1972551914\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-2tkgb-819368644\":{\"id\":\"pod-log-to-github-2tkgb-819368644\",\"name\":\"pod-log-to-github-2tkgb.root.upload-to-github.executor\",\"displayName\":\"executor\",\"type\":\"Pod\",\"templateName\":\"system-container-impl\",\"templateScope\":\"local/pod-log-to-github-2tkgb\",\"phase\":\"Failed\",\"boundaryID\":\"pod-log-to-github-2tkgb-969796849\",\"message\":\"Error (exit code 1)\",\"startedAt\":\"2025-05-07T05:36:55Z\",\"finishedAt\":\"2025-05-07T05:37:05Z\",\"progress\":\"0/1\",\"resourcesDuration\":{\"cpu\":17,\"memory\":17},\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"python:3.9\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"f994665a-c6dc-432d-a15f-07517f9c4f4d\\\",\\\"--execution_id\\\",\\\"32\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"report_file\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"name\\\\\\\":\\\\\\\"1\\\\\\\", \\\\\\\"type\\\\\\\":{\\\\\\\"instanceSchema\\\\\\\":\\\\\\\"\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/f994665a-c6dc-432d-a15f-07517f9c4f4d/collect-pod-logs/output_report\\\\\\\", \\\\\\\"metadata\\\\\\\":{\\\\\\\"display_name\\\\\\\":\\\\\\\"output_report\\\\\\\"}}]}}, \\\\\\\"parameterValues\\\\\\\":{\\\\\\\"branch\\\\\\\":\\\\\\\"main\\\\\\\", \\\\\\\"github_token\\\\\\\":\\\\\\\"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\\\\\\\", \\\\\\\"repo_url\\\\\\\":\\\\\\\"https://github.com/Rosa1026/Kubeflow.git\\\\\\\"}}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"inputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"report_file\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}, \\\\\\\"parameters\\\\\\\":{\\\\\\\"branch\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\", \\\\\\\"defaultValue\\\\\\\":\\\\\\\"main\\\\\\\", \\\\\\\"isOptional\\\\\\\":true}, \\\\\\\"github_token\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}, \\\\\\\"repo_url\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-upload-to-github\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str = \\\\\\\"main\\\\\\\"\\\\n):\\\\n    import os\\\\n    import subprocess\\\\n\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_file.path)\\\\n    if not os.path.exists(report_file.path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_file.path} does not exist\\\\\\\")\\\\n\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", report_file.path, os.path.join(repo_dir, \\\\\\\"report.md\\\\\\\")], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", \\\\\\\"Add report\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"resources\\\":{}}]}\"}]},\"outputs\":{\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-2tkgb/2025/05/07/pod-log-to-github-2tkgb-system-container-impl-819368644/main.log\"}}],\"exitCode\":\"1\"},\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-2tkgb-830918901\":{\"id\":\"pod-log-to-github-2tkgb-830918901\",\"name\":\"pod-log-to-github-2tkgb.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-2tkgb\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-2tkgb\",\"startedAt\":\"2025-05-07T05:36:04Z\",\"finishedAt\":\"2025-05-07T05:36:08Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":3,\"memory\":2},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-pod-logs\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-pod-logs\\\"},\\\"inputs\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"componentInputParameter\\\":\\\"namespace\\\"},\\\"pod_name\\\":{\\\"componentInputParameter\\\":\\\"pod_name\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-pod-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-pod-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-pod-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"namespace\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"pod_name\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\",\\\"github_token\\\":\\\"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\\\",\\\"namespace\\\":\\\"istio-system\\\",\\\"pod_name\\\":\\\"toolbox\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"value\":\"30\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"value\":\"0\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-2tkgb/2025/05/07/pod-log-to-github-2tkgb-system-dag-driver-830918901/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-2tkgb-2441106536\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-2tkgb-969796849\":{\"id\":\"pod-log-to-github-2tkgb-969796849\",\"name\":\"pod-log-to-github-2tkgb.root.upload-to-github\",\"displayName\":\"upload-to-github\",\"type\":\"DAG\",\"templateName\":\"system-container-executor\",\"templateScope\":\"local/pod-log-to-github-2tkgb\",\"phase\":\"Failed\",\"boundaryID\":\"pod-log-to-github-2tkgb-2441106536\",\"startedAt\":\"2025-05-07T05:36:55Z\",\"finishedAt\":\"2025-05-07T05:37:15Z\",\"progress\":\"0/1\",\"resourcesDuration\":{\"cpu\":17,\"memory\":17},\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"python:3.9\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"f994665a-c6dc-432d-a15f-07517f9c4f4d\\\",\\\"--execution_id\\\",\\\"32\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"report_file\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"name\\\\\\\":\\\\\\\"1\\\\\\\", \\\\\\\"type\\\\\\\":{\\\\\\\"instanceSchema\\\\\\\":\\\\\\\"\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/f994665a-c6dc-432d-a15f-07517f9c4f4d/collect-pod-logs/output_report\\\\\\\", \\\\\\\"metadata\\\\\\\":{\\\\\\\"display_name\\\\\\\":\\\\\\\"output_report\\\\\\\"}}]}}, \\\\\\\"parameterValues\\\\\\\":{\\\\\\\"branch\\\\\\\":\\\\\\\"main\\\\\\\", \\\\\\\"github_token\\\\\\\":\\\\\\\"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\\\\\\\", \\\\\\\"repo_url\\\\\\\":\\\\\\\"https://github.com/Rosa1026/Kubeflow.git\\\\\\\"}}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"inputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"report_file\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}, \\\\\\\"parameters\\\\\\\":{\\\\\\\"branch\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\", \\\\\\\"defaultValue\\\\\\\":\\\\\\\"main\\\\\\\", \\\\\\\"isOptional\\\\\\\":true}, \\\\\\\"github_token\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}, \\\\\\\"repo_url\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-upload-to-github\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str = \\\\\\\"main\\\\\\\"\\\\n):\\\\n    import os\\\\n    import subprocess\\\\n\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_file.path)\\\\n    if not os.path.exists(report_file.path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_file.path} does not exist\\\\\\\")\\\\n\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", report_file.path, os.path.join(repo_dir, \\\\\\\"report.md\\\\\\\")], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", \\\\\\\"Add report\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"resources\\\":{}}]}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"false\"}]},\"children\":[\"pod-log-to-github-2tkgb-819368644\"],\"outboundNodes\":[\"pod-log-to-github-2tkgb-819368644\"]}},\"conditions\":[{\"type\":\"PodRunning\",\"status\":\"False\"},{\"type\":\"Completed\",\"status\":\"True\"}],\"resourcesDuration\":{\"cpu\":41,\"memory\":38},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = NotFound desc = Failed to report workflow: NotFoundError: Failed to add PersistedFinalState label to workflow pod-log-to-github-2tkgb: workflows.argoproj.io \"pod-log-to-github-2tkgb\" not found: CustomError (code: 1): Error while reporting workflow resource (code: NotFound, message: Failed to report workflow: NotFoundError: Failed to add PersistedFinalState label to workflow pod-log-to-github-2tkgb: workflows.argoproj.io \"pod-log-to-github-2tkgb\" not found): rpc error: code = NotFound desc = Failed to report workflow: NotFoundError: Failed to add PersistedFinalState label to workflow pod-log-to-github-2tkgb: workflows.argoproj.io \"pod-log-to-github-2tkgb\" not found, {\"metadata\":{\"name\":\"pod-log-to-github-2tkgb\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"a5fd8c28-c792-45f3-9853-a321205997e9\",\"resourceVersion\":\"6048612\",\"generation\":9,\"creationTimestamp\":\"2025-05-07T05:36:04Z\",\"labels\":{\"pipeline/persistedFinalState\":\"true\",\"pipeline/runid\":\"f994665a-c6dc-432d-a15f-07517f9c4f4d\",\"workflows.argoproj.io/completed\":\"true\",\"workflows.argoproj.io/phase\":\"Failed\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-pod-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-pod-logs\\\",\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"pod_name\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}},\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-pod-logs\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-pod-logs\\\"},\\\"inputs\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"componentInputParameter\\\":\\\"namespace\\\"},\\\"pod_name\\\":{\\\"componentInputParameter\\\":\\\"pod_name\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-pod-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-pod-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-pod-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"namespace\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"pod_name\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-pod-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_pod_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_pod_logs(\\\\n    namespace: str,\\\\n    pod_name: str,\\\\n    output_report: Output[Dataset]\\\\n):\\\\n    import subprocess\\\\n\\\\n    describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n    logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n    events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n\\\\n    with open(output_report.path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(\\\\\\\"# 장애 복기 리포트\\\\\\\\n\\\\\\\")\\\\n        f.write(f\\\\\\\"**Pod**: {pod_name}\\\\\\\\n\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## describe\\\\\\\\n```text\\\\\\\\n\\\\\\\" + describe + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## logs\\\\\\\\n```text\\\\\\\\n\\\\\\\" + logs + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## events\\\\\\\\n```text\\\\\\\\n\\\\\\\" + events + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str = \\\\\\\"main\\\\\\\"\\\\n):\\\\n    import os\\\\n    import subprocess\\\\n\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_file.path)\\\\n    if not os.path.exists(report_file.path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_file.path} does not exist\\\\\\\")\\\\n\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", report_file.path, os.path.join(repo_dir, \\\\\\\"report.md\\\\\\\")], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", \\\\\\\"Add report\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Run of pod_log_pipeline_version_at_2025-05-07T05:35:37.141Z (40a7e)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-07T05:37:15Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/completed\":{},\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}},{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-07T05:48:49Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-pod-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-pod-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/persistedFinalState\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"f994665a-c6dc-432d-a15f-07517f9c4f4d\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-pod-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-pod-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-pod-logs\\\"},\\\"inputs\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"componentInputParameter\\\":\\\"namespace\\\"},\\\"pod_name\\\":{\\\"componentInputParameter\\\":\\\"pod_name\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-pod-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-pod-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-pod-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-pod-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-pod-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-pod-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-pod-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-pod-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-pod-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"f994665a-c6dc-432d-a15f-07517f9c4f4d\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\",\\\"github_token\\\":\\\"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\\\",\\\"namespace\\\":\\\"istio-system\\\",\\\"pod_name\\\":\\\"toolbox\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"f994665a-c6dc-432d-a15f-07517f9c4f4d\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Failed\",\"startedAt\":\"2025-05-07T05:36:04Z\",\"finishedAt\":\"2025-05-07T05:37:15Z\",\"progress\":\"4/5\",\"nodes\":{\"pod-log-to-github-2tkgb\":{\"id\":\"pod-log-to-github-2tkgb\",\"name\":\"pod-log-to-github-2tkgb\",\"displayName\":\"pod-log-to-github-2tkgb\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-2tkgb\",\"phase\":\"Failed\",\"startedAt\":\"2025-05-07T05:36:04Z\",\"finishedAt\":\"2025-05-07T05:37:15Z\",\"progress\":\"4/5\",\"resourcesDuration\":{\"cpu\":41,\"memory\":38},\"children\":[\"pod-log-to-github-2tkgb-830918901\"],\"outboundNodes\":[\"pod-log-to-github-2tkgb-819368644\"]},\"pod-log-to-github-2tkgb-1972551914\":{\"id\":\"pod-log-to-github-2tkgb-1972551914\",\"name\":\"pod-log-to-github-2tkgb.root.upload-to-github-driver\",\"displayName\":\"upload-to-github-driver\",\"type\":\"Pod\",\"templateName\":\"system-container-driver\",\"templateScope\":\"local/pod-log-to-github-2tkgb\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-2tkgb-2441106536\",\"startedAt\":\"2025-05-07T05:36:45Z\",\"finishedAt\":\"2025-05-07T05:36:48Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":4,\"memory\":3},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-pod-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-pod-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str = \\\\\\\"main\\\\\\\"\\\\n):\\\\n    import os\\\\n    import subprocess\\\\n\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_file.path)\\\\n    if not os.path.exists(report_file.path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_file.path} does not exist\\\\\\\")\\\\n\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", report_file.path, os.path.join(repo_dir, \\\\\\\"report.md\\\\\\\")], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", \\\\\\\"Add report\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\"},{\"name\":\"parent-dag-id\",\"value\":\"30\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\",\"value\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"python:3.9\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"f994665a-c6dc-432d-a15f-07517f9c4f4d\\\",\\\"--execution_id\\\",\\\"32\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"report_file\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"name\\\\\\\":\\\\\\\"1\\\\\\\", \\\\\\\"type\\\\\\\":{\\\\\\\"instanceSchema\\\\\\\":\\\\\\\"\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/f994665a-c6dc-432d-a15f-07517f9c4f4d/collect-pod-logs/output_report\\\\\\\", \\\\\\\"metadata\\\\\\\":{\\\\\\\"display_name\\\\\\\":\\\\\\\"output_report\\\\\\\"}}]}}, \\\\\\\"parameterValues\\\\\\\":{\\\\\\\"branch\\\\\\\":\\\\\\\"main\\\\\\\", \\\\\\\"github_token\\\\\\\":\\\\\\\"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\\\\\\\", \\\\\\\"repo_url\\\\\\\":\\\\\\\"https://github.com/Rosa1026/Kubeflow.git\\\\\\\"}}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"inputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"report_file\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}, \\\\\\\"parameters\\\\\\\":{\\\\\\\"branch\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\", \\\\\\\"defaultValue\\\\\\\":\\\\\\\"main\\\\\\\", \\\\\\\"isOptional\\\\\\\":true}, \\\\\\\"github_token\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}, \\\\\\\"repo_url\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-upload-to-github\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str = \\\\\\\"main\\\\\\\"\\\\n):\\\\n    import os\\\\n    import subprocess\\\\n\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_file.path)\\\\n    if not os.path.exists(report_file.path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_file.path} does not exist\\\\\\\")\\\\n\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", report_file.path, os.path.join(repo_dir, \\\\\\\"report.md\\\\\\\")], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", \\\\\\\"Add report\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"resources\\\":{}}]}\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-2tkgb/2025/05/07/pod-log-to-github-2tkgb-system-container-driver-1972551914/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-2tkgb-969796849\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-2tkgb-2441106536\":{\"id\":\"pod-log-to-github-2tkgb-2441106536\",\"name\":\"pod-log-to-github-2tkgb.root\",\"displayName\":\"root\",\"type\":\"DAG\",\"templateName\":\"root\",\"templateScope\":\"local/pod-log-to-github-2tkgb\",\"phase\":\"Failed\",\"boundaryID\":\"pod-log-to-github-2tkgb\",\"startedAt\":\"2025-05-07T05:36:14Z\",\"finishedAt\":\"2025-05-07T05:37:15Z\",\"progress\":\"3/4\",\"resourcesDuration\":{\"cpu\":38,\"memory\":36},\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"30\"}]},\"children\":[\"pod-log-to-github-2tkgb-3003919539\"],\"outboundNodes\":[\"pod-log-to-github-2tkgb-819368644\"]},\"pod-log-to-github-2tkgb-3003919539\":{\"id\":\"pod-log-to-github-2tkgb-3003919539\",\"name\":\"pod-log-to-github-2tkgb.root.collect-pod-logs-driver\",\"displayName\":\"collect-pod-logs-driver\",\"type\":\"Pod\",\"templateName\":\"system-container-driver\",\"templateScope\":\"local/pod-log-to-github-2tkgb\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-2tkgb-2441106536\",\"startedAt\":\"2025-05-07T05:36:14Z\",\"finishedAt\":\"2025-05-07T05:36:18Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":3,\"memory\":2},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"executorLabel\\\":\\\"exec-collect-pod-logs\\\",\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"pod_name\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}},\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-pod-logs\\\"},\\\"inputs\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"componentInputParameter\\\":\\\"namespace\\\"},\\\"pod_name\\\":{\\\"componentInputParameter\\\":\\\"pod_name\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-pod-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_pod_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_pod_logs(\\\\n    namespace: str,\\\\n    pod_name: str,\\\\n    output_report: Output[Dataset]\\\\n):\\\\n    import subprocess\\\\n\\\\n    describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n    logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n    events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n\\\\n    with open(output_report.path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(\\\\\\\"# 장애 복기 리포트\\\\\\\\n\\\\\\\")\\\\n        f.write(f\\\\\\\"**Pod**: {pod_name}\\\\\\\\n\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## describe\\\\\\\\n```text\\\\\\\\n\\\\\\\" + describe + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## logs\\\\\\\\n```text\\\\\\\\n\\\\\\\" + logs + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## events\\\\\\\\n```text\\\\\\\\n\\\\\\\" + events + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\"},{\"name\":\"parent-dag-id\",\"value\":\"30\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\",\"value\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"python:3.9\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"f994665a-c6dc-432d-a15f-07517f9c4f4d\\\",\\\"--execution_id\\\",\\\"31\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{\\\\\\\"parameterValues\\\\\\\":{\\\\\\\"namespace\\\\\\\":\\\\\\\"istio-system\\\\\\\", \\\\\\\"pod_name\\\\\\\":\\\\\\\"toolbox\\\\\\\"}}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/f994665a-c6dc-432d-a15f-07517f9c4f4d/collect-pod-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"inputDefinitions\\\\\\\":{\\\\\\\"parameters\\\\\\\":{\\\\\\\"namespace\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}, \\\\\\\"pod_name\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}}}, \\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-pod-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_pod_logs(\\\\n    namespace: str,\\\\n    pod_name: str,\\\\n    output_report: Output[Dataset]\\\\n):\\\\n    import subprocess\\\\n\\\\n    describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n    logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n    events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n\\\\n    with open(output_report.path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(\\\\\\\"# 장애 복기 리포트\\\\\\\\n\\\\\\\")\\\\n        f.write(f\\\\\\\"**Pod**: {pod_name}\\\\\\\\n\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## describe\\\\\\\\n```text\\\\\\\\n\\\\\\\" + describe + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## logs\\\\\\\\n```text\\\\\\\\n\\\\\\\" + logs + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## events\\\\\\\\n```text\\\\\\\\n\\\\\\\" + events + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_pod_logs\\\"],\\\"resources\\\":{}}]}\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-2tkgb/2025/05/07/pod-log-to-github-2tkgb-system-container-driver-3003919539/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-2tkgb-3928305142\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-2tkgb-3928305142\":{\"id\":\"pod-log-to-github-2tkgb-3928305142\",\"name\":\"pod-log-to-github-2tkgb.root.collect-pod-logs\",\"displayName\":\"collect-pod-logs\",\"type\":\"DAG\",\"templateName\":\"system-container-executor\",\"templateScope\":\"local/pod-log-to-github-2tkgb\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-2tkgb-2441106536\",\"startedAt\":\"2025-05-07T05:36:24Z\",\"finishedAt\":\"2025-05-07T05:36:45Z\",\"progress\":\"2/3\",\"resourcesDuration\":{\"cpu\":35,\"memory\":34},\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"python:3.9\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"f994665a-c6dc-432d-a15f-07517f9c4f4d\\\",\\\"--execution_id\\\",\\\"31\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{\\\\\\\"parameterValues\\\\\\\":{\\\\\\\"namespace\\\\\\\":\\\\\\\"istio-system\\\\\\\", \\\\\\\"pod_name\\\\\\\":\\\\\\\"toolbox\\\\\\\"}}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/f994665a-c6dc-432d-a15f-07517f9c4f4d/collect-pod-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"inputDefinitions\\\\\\\":{\\\\\\\"parameters\\\\\\\":{\\\\\\\"namespace\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}, \\\\\\\"pod_name\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}}}, \\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-pod-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_pod_logs(\\\\n    namespace: str,\\\\n    pod_name: str,\\\\n    output_report: Output[Dataset]\\\\n):\\\\n    import subprocess\\\\n\\\\n    describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n    logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n    events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n\\\\n    with open(output_report.path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(\\\\\\\"# 장애 복기 리포트\\\\\\\\n\\\\\\\")\\\\n        f.write(f\\\\\\\"**Pod**: {pod_name}\\\\\\\\n\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## describe\\\\\\\\n```text\\\\\\\\n\\\\\\\" + describe + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## logs\\\\\\\\n```text\\\\\\\\n\\\\\\\" + logs + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## events\\\\\\\\n```text\\\\\\\\n\\\\\\\" + events + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_pod_logs\\\"],\\\"resources\\\":{}}]}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"false\"}]},\"children\":[\"pod-log-to-github-2tkgb-610323093\"],\"outboundNodes\":[\"pod-log-to-github-2tkgb-610323093\"]},\"pod-log-to-github-2tkgb-610323093\":{\"id\":\"pod-log-to-github-2tkgb-610323093\",\"name\":\"pod-log-to-github-2tkgb.root.collect-pod-logs.executor\",\"displayName\":\"executor\",\"type\":\"Pod\",\"templateName\":\"system-container-impl\",\"templateScope\":\"local/pod-log-to-github-2tkgb\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-2tkgb-3928305142\",\"startedAt\":\"2025-05-07T05:36:24Z\",\"finishedAt\":\"2025-05-07T05:36:34Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":14,\"memory\":14},\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"python:3.9\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"f994665a-c6dc-432d-a15f-07517f9c4f4d\\\",\\\"--execution_id\\\",\\\"31\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{\\\\\\\"parameterValues\\\\\\\":{\\\\\\\"namespace\\\\\\\":\\\\\\\"istio-system\\\\\\\", \\\\\\\"pod_name\\\\\\\":\\\\\\\"toolbox\\\\\\\"}}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/f994665a-c6dc-432d-a15f-07517f9c4f4d/collect-pod-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"inputDefinitions\\\\\\\":{\\\\\\\"parameters\\\\\\\":{\\\\\\\"namespace\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}, \\\\\\\"pod_name\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}}}, \\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-pod-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_pod_logs(\\\\n    namespace: str,\\\\n    pod_name: str,\\\\n    output_report: Output[Dataset]\\\\n):\\\\n    import subprocess\\\\n\\\\n    describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n    logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n    events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n\\\\n    with open(output_report.path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(\\\\\\\"# 장애 복기 리포트\\\\\\\\n\\\\\\\")\\\\n        f.write(f\\\\\\\"**Pod**: {pod_name}\\\\\\\\n\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## describe\\\\\\\\n```text\\\\\\\\n\\\\\\\" + describe + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## logs\\\\\\\\n```text\\\\\\\\n\\\\\\\" + logs + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## events\\\\\\\\n```text\\\\\\\\n\\\\\\\" + events + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_pod_logs\\\"],\\\"resources\\\":{}}]}\"}]},\"outputs\":{\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-2tkgb/2025/05/07/pod-log-to-github-2tkgb-system-container-impl-610323093/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-2tkgb-1972551914\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-2tkgb-819368644\":{\"id\":\"pod-log-to-github-2tkgb-819368644\",\"name\":\"pod-log-to-github-2tkgb.root.upload-to-github.executor\",\"displayName\":\"executor\",\"type\":\"Pod\",\"templateName\":\"system-container-impl\",\"templateScope\":\"local/pod-log-to-github-2tkgb\",\"phase\":\"Failed\",\"boundaryID\":\"pod-log-to-github-2tkgb-969796849\",\"message\":\"Error (exit code 1)\",\"startedAt\":\"2025-05-07T05:36:55Z\",\"finishedAt\":\"2025-05-07T05:37:05Z\",\"progress\":\"0/1\",\"resourcesDuration\":{\"cpu\":17,\"memory\":17},\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"python:3.9\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"f994665a-c6dc-432d-a15f-07517f9c4f4d\\\",\\\"--execution_id\\\",\\\"32\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"report_file\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"name\\\\\\\":\\\\\\\"1\\\\\\\", \\\\\\\"type\\\\\\\":{\\\\\\\"instanceSchema\\\\\\\":\\\\\\\"\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/f994665a-c6dc-432d-a15f-07517f9c4f4d/collect-pod-logs/output_report\\\\\\\", \\\\\\\"metadata\\\\\\\":{\\\\\\\"display_name\\\\\\\":\\\\\\\"output_report\\\\\\\"}}]}}, \\\\\\\"parameterValues\\\\\\\":{\\\\\\\"branch\\\\\\\":\\\\\\\"main\\\\\\\", \\\\\\\"github_token\\\\\\\":\\\\\\\"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\\\\\\\", \\\\\\\"repo_url\\\\\\\":\\\\\\\"https://github.com/Rosa1026/Kubeflow.git\\\\\\\"}}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"inputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"report_file\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}, \\\\\\\"parameters\\\\\\\":{\\\\\\\"branch\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\", \\\\\\\"defaultValue\\\\\\\":\\\\\\\"main\\\\\\\", \\\\\\\"isOptional\\\\\\\":true}, \\\\\\\"github_token\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}, \\\\\\\"repo_url\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-upload-to-github\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str = \\\\\\\"main\\\\\\\"\\\\n):\\\\n    import os\\\\n    import subprocess\\\\n\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_file.path)\\\\n    if not os.path.exists(report_file.path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_file.path} does not exist\\\\\\\")\\\\n\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", report_file.path, os.path.join(repo_dir, \\\\\\\"report.md\\\\\\\")], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", \\\\\\\"Add report\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"resources\\\":{}}]}\"}]},\"outputs\":{\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-2tkgb/2025/05/07/pod-log-to-github-2tkgb-system-container-impl-819368644/main.log\"}}],\"exitCode\":\"1\"},\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-2tkgb-830918901\":{\"id\":\"pod-log-to-github-2tkgb-830918901\",\"name\":\"pod-log-to-github-2tkgb.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-2tkgb\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-2tkgb\",\"startedAt\":\"2025-05-07T05:36:04Z\",\"finishedAt\":\"2025-05-07T05:36:08Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":3,\"memory\":2},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-pod-logs\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-pod-logs\\\"},\\\"inputs\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"componentInputParameter\\\":\\\"namespace\\\"},\\\"pod_name\\\":{\\\"componentInputParameter\\\":\\\"pod_name\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-pod-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-pod-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-pod-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"namespace\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"pod_name\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\",\\\"github_token\\\":\\\"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\\\",\\\"namespace\\\":\\\"istio-system\\\",\\\"pod_name\\\":\\\"toolbox\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"value\":\"30\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"value\":\"0\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-2tkgb/2025/05/07/pod-log-to-github-2tkgb-system-dag-driver-830918901/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-2tkgb-2441106536\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-2tkgb-969796849\":{\"id\":\"pod-log-to-github-2tkgb-969796849\",\"name\":\"pod-log-to-github-2tkgb.root.upload-to-github\",\"displayName\":\"upload-to-github\",\"type\":\"DAG\",\"templateName\":\"system-container-executor\",\"templateScope\":\"local/pod-log-to-github-2tkgb\",\"phase\":\"Failed\",\"boundaryID\":\"pod-log-to-github-2tkgb-2441106536\",\"startedAt\":\"2025-05-07T05:36:55Z\",\"finishedAt\":\"2025-05-07T05:37:15Z\",\"progress\":\"0/1\",\"resourcesDuration\":{\"cpu\":17,\"memory\":17},\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"python:3.9\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"f994665a-c6dc-432d-a15f-07517f9c4f4d\\\",\\\"--execution_id\\\",\\\"32\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"report_file\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"name\\\\\\\":\\\\\\\"1\\\\\\\", \\\\\\\"type\\\\\\\":{\\\\\\\"instanceSchema\\\\\\\":\\\\\\\"\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/f994665a-c6dc-432d-a15f-07517f9c4f4d/collect-pod-logs/output_report\\\\\\\", \\\\\\\"metadata\\\\\\\":{\\\\\\\"display_name\\\\\\\":\\\\\\\"output_report\\\\\\\"}}]}}, \\\\\\\"parameterValues\\\\\\\":{\\\\\\\"branch\\\\\\\":\\\\\\\"main\\\\\\\", \\\\\\\"github_token\\\\\\\":\\\\\\\"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\\\\\\\", \\\\\\\"repo_url\\\\\\\":\\\\\\\"https://github.com/Rosa1026/Kubeflow.git\\\\\\\"}}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"inputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"report_file\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}, \\\\\\\"parameters\\\\\\\":{\\\\\\\"branch\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\", \\\\\\\"defaultValue\\\\\\\":\\\\\\\"main\\\\\\\", \\\\\\\"isOptional\\\\\\\":true}, \\\\\\\"github_token\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}, \\\\\\\"repo_url\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-upload-to-github\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str = \\\\\\\"main\\\\\\\"\\\\n):\\\\n    import os\\\\n    import subprocess\\\\n\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_file.path)\\\\n    if not os.path.exists(report_file.path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_file.path} does not exist\\\\\\\")\\\\n\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", report_file.path, os.path.join(repo_dir, \\\\\\\"report.md\\\\\\\")], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", \\\\\\\"Add report\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"resources\\\":{}}]}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"false\"}]},\"children\":[\"pod-log-to-github-2tkgb-819368644\"],\"outboundNodes\":[\"pod-log-to-github-2tkgb-819368644\"]}},\"conditions\":[{\"type\":\"PodRunning\",\"status\":\"False\"},{\"type\":\"Completed\",\"status\":\"True\"}],\"resourcesDuration\":{\"cpu\":41,\"memory\":38},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = NotFound desc = Failed to report workflow: NotFoundError: Failed to add PersistedFinalState label to workflow pod-log-to-github-2tkgb: workflows.argoproj.io \"pod-log-to-github-2tkgb\" not found"
time="2025-05-08T05:37:23Z" level=error msg="Permanent failure while syncing resource (kubeflow-user-example-com/pod-log-to-github-2tkgb): CustomError (code: 1): Workflow (kubeflow-user-example-com/pod-log-to-github-2tkgb) in work queue no longer exists: CustomError (code: 2): Error retrieving workflow (pod-log-to-github-2tkgb) in namespace (kubeflow-user-example-com): Error retrieving workflow (pod-log-to-github-2tkgb) in namespace (kubeflow-user-example-com): workflow.argoproj.io \"pod-log-to-github-2tkgb\" not found: workflow.argoproj.io \"pod-log-to-github-2tkgb\" not found: Error retrieving workflow (pod-log-to-github-2tkgb) in namespace (kubeflow-user-example-com): workflow.argoproj.io \"pod-log-to-github-2tkgb\" not found: workflow.argoproj.io \"pod-log-to-github-2tkgb\" not found: CustomError (code: 2): Error retrieving workflow (pod-log-to-github-2tkgb) in namespace (kubeflow-user-example-com): Error retrieving workflow (pod-log-to-github-2tkgb) in namespace (kubeflow-user-example-com): workflow.argoproj.io \"pod-log-to-github-2tkgb\" not found: workflow.argoproj.io \"pod-log-to-github-2tkgb\" not found: Error retrieving workflow (pod-log-to-github-2tkgb) in namespace (kubeflow-user-example-com): workflow.argoproj.io \"pod-log-to-github-2tkgb\" not found: workflow.argoproj.io \"pod-log-to-github-2tkgb\" not found"
time="2025-05-08T05:41:07Z" level=error msg="Transient failure while syncing resource (kubeflow-user-example-com/pod-log-to-github-zj76w): CustomError (code: 0): Syncing Workflow (pod-log-to-github-zj76w): transient failure: CustomError (code: 0): Error while reporting workflow resource (code: Unauthenticated, message: Jwt is expired): rpc error: code = Unauthenticated desc = Jwt is expired, {\"metadata\":{\"name\":\"pod-log-to-github-zj76w\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"3066c70d-3732-4a31-830b-af01907bb8cd\",\"resourceVersion\":\"6535331\",\"generation\":2,\"creationTimestamp\":\"2025-05-08T05:41:06Z\",\"labels\":{\"pipeline/runid\":\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"workflows.argoproj.io/phase\":\"Running\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-filtered-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"defaultValue\\\":\\\"crash\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"}}},\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\\\\\"crash\\\\\\\"):\\\\n    import shutil\\\\n    import json\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # kubectl 설치 방법 변경\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([ \\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods in all namespaces\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(f\\\\\\\"\\\\u003chtml\\\\u003e\\\\u003cbody\\\\u003e\\\\u003ch1\\\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\\\u003c/h1\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in describe.splitlines() if keyword in l])\\\\n            matched_logs = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in logs.splitlines() if keyword in l])\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003ch2\\\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\\\u003c/h2\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eDescribe\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_describe}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eLogs\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_logs}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in events.splitlines() if keyword in l])\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        # Write filtered events grouped by namespace\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003ch3\\\\u003eEvents in Namespace: `{ns}`\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"'  \\\\u0026\\\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str = \\\\\\\"main\\\\\\\"\\\\n):\\\\n\\\\n    import subprocess\\\\n    import shutil\\\\n    import os\\\\n    from datetime import datetime\\\\n\\\\n    os.environ[\\\\\\\"GIT_ASKPASS\\\\\\\"] = \\\\\\\"echo\\\\\\\"\\\\n\\\\n    report_path = report_file.path\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_path)\\\\n    if not os.path.exists(report_path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_path} does not exist\\\\\\\")\\\\n\\\\n    # HTML 파일 이름 만들기\\\\n    timestamp = datetime.now().strftime(\\\\\\\"report-%Y%m%d-%H%M%S.html\\\\\\\")\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n    report_target_path = os.path.join(repo_dir, timestamp)\\\\n\\\\n    # Git clone\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    shutil.copyfile(report_path, report_target_path)\\\\n\\\\n    # Git config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    # Git commit \\\\u0026 push\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", f\\\\\\\"Add {timestamp}\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(f\\\\\\\"Report committed and pushed to GitHub as {timestamp}.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Clone of Run of pod_log_pipeline_advance_version_at_2025-05-08T05:25:35.947Z (05c63)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T05:41:06Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}},{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T05:41:06Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-filtered-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-filtered-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-filtered-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-filtered-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-filtered-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-filtered-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Running\",\"startedAt\":\"2025-05-08T05:41:06Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"nodes\":{\"pod-log-to-github-zj76w\":{\"id\":\"pod-log-to-github-zj76w\",\"name\":\"pod-log-to-github-zj76w\",\"displayName\":\"pod-log-to-github-zj76w\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-zj76w\",\"phase\":\"Running\",\"startedAt\":\"2025-05-08T05:41:06Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"children\":[\"pod-log-to-github-zj76w-1372256085\"]},\"pod-log-to-github-zj76w-1372256085\":{\"id\":\"pod-log-to-github-zj76w-1372256085\",\"name\":\"pod-log-to-github-zj76w.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-zj76w\",\"phase\":\"Pending\",\"boundaryID\":\"pod-log-to-github-zj76w\",\"startedAt\":\"2025-05-08T05:41:06Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]}}},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = Unauthenticated desc = Jwt is expired: CustomError (code: 0): Error while reporting workflow resource (code: Unauthenticated, message: Jwt is expired): rpc error: code = Unauthenticated desc = Jwt is expired, {\"metadata\":{\"name\":\"pod-log-to-github-zj76w\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"3066c70d-3732-4a31-830b-af01907bb8cd\",\"resourceVersion\":\"6535331\",\"generation\":2,\"creationTimestamp\":\"2025-05-08T05:41:06Z\",\"labels\":{\"pipeline/runid\":\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"workflows.argoproj.io/phase\":\"Running\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-filtered-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"defaultValue\\\":\\\"crash\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"}}},\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\\\\\"crash\\\\\\\"):\\\\n    import shutil\\\\n    import json\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # kubectl 설치 방법 변경\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([ \\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods in all namespaces\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(f\\\\\\\"\\\\u003chtml\\\\u003e\\\\u003cbody\\\\u003e\\\\u003ch1\\\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\\\u003c/h1\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in describe.splitlines() if keyword in l])\\\\n            matched_logs = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in logs.splitlines() if keyword in l])\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003ch2\\\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\\\u003c/h2\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eDescribe\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_describe}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eLogs\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_logs}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in events.splitlines() if keyword in l])\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        # Write filtered events grouped by namespace\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003ch3\\\\u003eEvents in Namespace: `{ns}`\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"'  \\\\u0026\\\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str = \\\\\\\"main\\\\\\\"\\\\n):\\\\n\\\\n    import subprocess\\\\n    import shutil\\\\n    import os\\\\n    from datetime import datetime\\\\n\\\\n    os.environ[\\\\\\\"GIT_ASKPASS\\\\\\\"] = \\\\\\\"echo\\\\\\\"\\\\n\\\\n    report_path = report_file.path\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_path)\\\\n    if not os.path.exists(report_path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_path} does not exist\\\\\\\")\\\\n\\\\n    # HTML 파일 이름 만들기\\\\n    timestamp = datetime.now().strftime(\\\\\\\"report-%Y%m%d-%H%M%S.html\\\\\\\")\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n    report_target_path = os.path.join(repo_dir, timestamp)\\\\n\\\\n    # Git clone\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    shutil.copyfile(report_path, report_target_path)\\\\n\\\\n    # Git config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    # Git commit \\\\u0026 push\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", f\\\\\\\"Add {timestamp}\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(f\\\\\\\"Report committed and pushed to GitHub as {timestamp}.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Clone of Run of pod_log_pipeline_advance_version_at_2025-05-08T05:25:35.947Z (05c63)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T05:41:06Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}},{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T05:41:06Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-filtered-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-filtered-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-filtered-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-filtered-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-filtered-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-filtered-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Running\",\"startedAt\":\"2025-05-08T05:41:06Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"nodes\":{\"pod-log-to-github-zj76w\":{\"id\":\"pod-log-to-github-zj76w\",\"name\":\"pod-log-to-github-zj76w\",\"displayName\":\"pod-log-to-github-zj76w\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-zj76w\",\"phase\":\"Running\",\"startedAt\":\"2025-05-08T05:41:06Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"children\":[\"pod-log-to-github-zj76w-1372256085\"]},\"pod-log-to-github-zj76w-1372256085\":{\"id\":\"pod-log-to-github-zj76w-1372256085\",\"name\":\"pod-log-to-github-zj76w.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-zj76w\",\"phase\":\"Pending\",\"boundaryID\":\"pod-log-to-github-zj76w\",\"startedAt\":\"2025-05-08T05:41:06Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]}}},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = Unauthenticated desc = Jwt is expired"
time="2025-05-08T05:41:11Z" level=error msg="Transient failure while syncing resource (kubeflow-user-example-com/pod-log-to-github-zj76w): CustomError (code: 0): Syncing Workflow (pod-log-to-github-zj76w): transient failure: CustomError (code: 0): Error while reporting workflow resource (code: Unauthenticated, message: Jwt is expired): rpc error: code = Unauthenticated desc = Jwt is expired, {\"metadata\":{\"name\":\"pod-log-to-github-zj76w\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"3066c70d-3732-4a31-830b-af01907bb8cd\",\"resourceVersion\":\"6535331\",\"generation\":2,\"creationTimestamp\":\"2025-05-08T05:41:06Z\",\"labels\":{\"pipeline/runid\":\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"workflows.argoproj.io/phase\":\"Running\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-filtered-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"defaultValue\\\":\\\"crash\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"}}},\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\\\\\"crash\\\\\\\"):\\\\n    import shutil\\\\n    import json\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # kubectl 설치 방법 변경\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([ \\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods in all namespaces\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(f\\\\\\\"\\\\u003chtml\\\\u003e\\\\u003cbody\\\\u003e\\\\u003ch1\\\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\\\u003c/h1\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in describe.splitlines() if keyword in l])\\\\n            matched_logs = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in logs.splitlines() if keyword in l])\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003ch2\\\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\\\u003c/h2\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eDescribe\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_describe}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eLogs\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_logs}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in events.splitlines() if keyword in l])\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        # Write filtered events grouped by namespace\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003ch3\\\\u003eEvents in Namespace: `{ns}`\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"'  \\\\u0026\\\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str = \\\\\\\"main\\\\\\\"\\\\n):\\\\n\\\\n    import subprocess\\\\n    import shutil\\\\n    import os\\\\n    from datetime import datetime\\\\n\\\\n    os.environ[\\\\\\\"GIT_ASKPASS\\\\\\\"] = \\\\\\\"echo\\\\\\\"\\\\n\\\\n    report_path = report_file.path\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_path)\\\\n    if not os.path.exists(report_path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_path} does not exist\\\\\\\")\\\\n\\\\n    # HTML 파일 이름 만들기\\\\n    timestamp = datetime.now().strftime(\\\\\\\"report-%Y%m%d-%H%M%S.html\\\\\\\")\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n    report_target_path = os.path.join(repo_dir, timestamp)\\\\n\\\\n    # Git clone\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    shutil.copyfile(report_path, report_target_path)\\\\n\\\\n    # Git config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    # Git commit \\\\u0026 push\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", f\\\\\\\"Add {timestamp}\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(f\\\\\\\"Report committed and pushed to GitHub as {timestamp}.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Clone of Run of pod_log_pipeline_advance_version_at_2025-05-08T05:25:35.947Z (05c63)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T05:41:06Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}},{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T05:41:06Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-filtered-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-filtered-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-filtered-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-filtered-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-filtered-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-filtered-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Running\",\"startedAt\":\"2025-05-08T05:41:06Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"nodes\":{\"pod-log-to-github-zj76w\":{\"id\":\"pod-log-to-github-zj76w\",\"name\":\"pod-log-to-github-zj76w\",\"displayName\":\"pod-log-to-github-zj76w\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-zj76w\",\"phase\":\"Running\",\"startedAt\":\"2025-05-08T05:41:06Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"children\":[\"pod-log-to-github-zj76w-1372256085\"]},\"pod-log-to-github-zj76w-1372256085\":{\"id\":\"pod-log-to-github-zj76w-1372256085\",\"name\":\"pod-log-to-github-zj76w.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-zj76w\",\"phase\":\"Pending\",\"boundaryID\":\"pod-log-to-github-zj76w\",\"startedAt\":\"2025-05-08T05:41:06Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]}}},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = Unauthenticated desc = Jwt is expired: CustomError (code: 0): Error while reporting workflow resource (code: Unauthenticated, message: Jwt is expired): rpc error: code = Unauthenticated desc = Jwt is expired, {\"metadata\":{\"name\":\"pod-log-to-github-zj76w\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"3066c70d-3732-4a31-830b-af01907bb8cd\",\"resourceVersion\":\"6535331\",\"generation\":2,\"creationTimestamp\":\"2025-05-08T05:41:06Z\",\"labels\":{\"pipeline/runid\":\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"workflows.argoproj.io/phase\":\"Running\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-filtered-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"defaultValue\\\":\\\"crash\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"}}},\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\\\\\"crash\\\\\\\"):\\\\n    import shutil\\\\n    import json\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # kubectl 설치 방법 변경\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([ \\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods in all namespaces\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(f\\\\\\\"\\\\u003chtml\\\\u003e\\\\u003cbody\\\\u003e\\\\u003ch1\\\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\\\u003c/h1\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in describe.splitlines() if keyword in l])\\\\n            matched_logs = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in logs.splitlines() if keyword in l])\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003ch2\\\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\\\u003c/h2\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eDescribe\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_describe}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eLogs\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_logs}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in events.splitlines() if keyword in l])\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        # Write filtered events grouped by namespace\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003ch3\\\\u003eEvents in Namespace: `{ns}`\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"'  \\\\u0026\\\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str = \\\\\\\"main\\\\\\\"\\\\n):\\\\n\\\\n    import subprocess\\\\n    import shutil\\\\n    import os\\\\n    from datetime import datetime\\\\n\\\\n    os.environ[\\\\\\\"GIT_ASKPASS\\\\\\\"] = \\\\\\\"echo\\\\\\\"\\\\n\\\\n    report_path = report_file.path\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_path)\\\\n    if not os.path.exists(report_path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_path} does not exist\\\\\\\")\\\\n\\\\n    # HTML 파일 이름 만들기\\\\n    timestamp = datetime.now().strftime(\\\\\\\"report-%Y%m%d-%H%M%S.html\\\\\\\")\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n    report_target_path = os.path.join(repo_dir, timestamp)\\\\n\\\\n    # Git clone\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    shutil.copyfile(report_path, report_target_path)\\\\n\\\\n    # Git config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    # Git commit \\\\u0026 push\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", f\\\\\\\"Add {timestamp}\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(f\\\\\\\"Report committed and pushed to GitHub as {timestamp}.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Clone of Run of pod_log_pipeline_advance_version_at_2025-05-08T05:25:35.947Z (05c63)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T05:41:06Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}},{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T05:41:06Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-filtered-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-filtered-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-filtered-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-filtered-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-filtered-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-filtered-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Running\",\"startedAt\":\"2025-05-08T05:41:06Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"nodes\":{\"pod-log-to-github-zj76w\":{\"id\":\"pod-log-to-github-zj76w\",\"name\":\"pod-log-to-github-zj76w\",\"displayName\":\"pod-log-to-github-zj76w\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-zj76w\",\"phase\":\"Running\",\"startedAt\":\"2025-05-08T05:41:06Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"children\":[\"pod-log-to-github-zj76w-1372256085\"]},\"pod-log-to-github-zj76w-1372256085\":{\"id\":\"pod-log-to-github-zj76w-1372256085\",\"name\":\"pod-log-to-github-zj76w.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-zj76w\",\"phase\":\"Pending\",\"boundaryID\":\"pod-log-to-github-zj76w\",\"startedAt\":\"2025-05-08T05:41:06Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]}}},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = Unauthenticated desc = Jwt is expired"
time="2025-05-08T05:41:19Z" level=error msg="Transient failure while syncing resource (kubeflow-user-example-com/pod-log-to-github-zj76w): CustomError (code: 0): Syncing Workflow (pod-log-to-github-zj76w): transient failure: CustomError (code: 0): Error while reporting workflow resource (code: Unauthenticated, message: Jwt is expired): rpc error: code = Unauthenticated desc = Jwt is expired, {\"metadata\":{\"name\":\"pod-log-to-github-zj76w\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"3066c70d-3732-4a31-830b-af01907bb8cd\",\"resourceVersion\":\"6535410\",\"generation\":3,\"creationTimestamp\":\"2025-05-08T05:41:06Z\",\"labels\":{\"pipeline/runid\":\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"workflows.argoproj.io/phase\":\"Running\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-filtered-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"defaultValue\\\":\\\"crash\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"}}},\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\\\\\"crash\\\\\\\"):\\\\n    import shutil\\\\n    import json\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # kubectl 설치 방법 변경\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([ \\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods in all namespaces\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(f\\\\\\\"\\\\u003chtml\\\\u003e\\\\u003cbody\\\\u003e\\\\u003ch1\\\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\\\u003c/h1\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in describe.splitlines() if keyword in l])\\\\n            matched_logs = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in logs.splitlines() if keyword in l])\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003ch2\\\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\\\u003c/h2\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eDescribe\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_describe}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eLogs\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_logs}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in events.splitlines() if keyword in l])\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        # Write filtered events grouped by namespace\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003ch3\\\\u003eEvents in Namespace: `{ns}`\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"'  \\\\u0026\\\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str = \\\\\\\"main\\\\\\\"\\\\n):\\\\n\\\\n    import subprocess\\\\n    import shutil\\\\n    import os\\\\n    from datetime import datetime\\\\n\\\\n    os.environ[\\\\\\\"GIT_ASKPASS\\\\\\\"] = \\\\\\\"echo\\\\\\\"\\\\n\\\\n    report_path = report_file.path\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_path)\\\\n    if not os.path.exists(report_path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_path} does not exist\\\\\\\")\\\\n\\\\n    # HTML 파일 이름 만들기\\\\n    timestamp = datetime.now().strftime(\\\\\\\"report-%Y%m%d-%H%M%S.html\\\\\\\")\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n    report_target_path = os.path.join(repo_dir, timestamp)\\\\n\\\\n    # Git clone\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    shutil.copyfile(report_path, report_target_path)\\\\n\\\\n    # Git config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    # Git commit \\\\u0026 push\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", f\\\\\\\"Add {timestamp}\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(f\\\\\\\"Report committed and pushed to GitHub as {timestamp}.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Clone of Run of pod_log_pipeline_advance_version_at_2025-05-08T05:25:35.947Z (05c63)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T05:41:06Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}},{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T05:41:16Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-filtered-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-filtered-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-filtered-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-filtered-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-filtered-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-filtered-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Running\",\"startedAt\":\"2025-05-08T05:41:06Z\",\"finishedAt\":null,\"progress\":\"1/2\",\"nodes\":{\"pod-log-to-github-zj76w\":{\"id\":\"pod-log-to-github-zj76w\",\"name\":\"pod-log-to-github-zj76w\",\"displayName\":\"pod-log-to-github-zj76w\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-zj76w\",\"phase\":\"Running\",\"startedAt\":\"2025-05-08T05:41:06Z\",\"finishedAt\":null,\"progress\":\"1/2\",\"children\":[\"pod-log-to-github-zj76w-1372256085\"]},\"pod-log-to-github-zj76w-1372256085\":{\"id\":\"pod-log-to-github-zj76w-1372256085\",\"name\":\"pod-log-to-github-zj76w.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-zj76w\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-zj76w\",\"startedAt\":\"2025-05-08T05:41:06Z\",\"finishedAt\":\"2025-05-08T05:41:10Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":3,\"memory\":2},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"value\":\"125\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"value\":\"0\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-zj76w/2025/05/08/pod-log-to-github-zj76w-system-dag-driver-1372256085/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-zj76w-402210888\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-zj76w-402210888\":{\"id\":\"pod-log-to-github-zj76w-402210888\",\"name\":\"pod-log-to-github-zj76w.root\",\"displayName\":\"root\",\"type\":\"DAG\",\"templateName\":\"root\",\"templateScope\":\"local/pod-log-to-github-zj76w\",\"phase\":\"Running\",\"boundaryID\":\"pod-log-to-github-zj76w\",\"startedAt\":\"2025-05-08T05:41:16Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"125\"}]},\"children\":[\"pod-log-to-github-zj76w-878217725\"]},\"pod-log-to-github-zj76w-878217725\":{\"id\":\"pod-log-to-github-zj76w-878217725\",\"name\":\"pod-log-to-github-zj76w.root.collect-filtered-logs-driver\",\"displayName\":\"collect-filtered-logs-driver\",\"type\":\"Pod\",\"templateName\":\"system-container-driver\",\"templateScope\":\"local/pod-log-to-github-zj76w\",\"phase\":\"Pending\",\"boundaryID\":\"pod-log-to-github-zj76w-402210888\",\"startedAt\":\"2025-05-08T05:41:16Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"defaultValue\\\":\\\"crash\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"}}},\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\\\\\"crash\\\\\\\"):\\\\n    import shutil\\\\n    import json\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # kubectl 설치 방법 변경\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([ \\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods in all namespaces\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(f\\\\\\\"\\\\u003chtml\\\\u003e\\\\u003cbody\\\\u003e\\\\u003ch1\\\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\\\u003c/h1\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in describe.splitlines() if keyword in l])\\\\n            matched_logs = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in logs.splitlines() if keyword in l])\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003ch2\\\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\\\u003c/h2\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eDescribe\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_describe}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eLogs\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_logs}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in events.splitlines() if keyword in l])\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        # Write filtered events grouped by namespace\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003ch3\\\\u003eEvents in Namespace: `{ns}`\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\"},{\"name\":\"parent-dag-id\",\"value\":\"125\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\",\"value\":\"\"}]}}},\"conditions\":[{\"type\":\"PodRunning\",\"status\":\"False\"}],\"resourcesDuration\":{\"cpu\":3,\"memory\":2},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = Unauthenticated desc = Jwt is expired: CustomError (code: 0): Error while reporting workflow resource (code: Unauthenticated, message: Jwt is expired): rpc error: code = Unauthenticated desc = Jwt is expired, {\"metadata\":{\"name\":\"pod-log-to-github-zj76w\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"3066c70d-3732-4a31-830b-af01907bb8cd\",\"resourceVersion\":\"6535410\",\"generation\":3,\"creationTimestamp\":\"2025-05-08T05:41:06Z\",\"labels\":{\"pipeline/runid\":\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"workflows.argoproj.io/phase\":\"Running\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-filtered-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"defaultValue\\\":\\\"crash\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"}}},\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\\\\\"crash\\\\\\\"):\\\\n    import shutil\\\\n    import json\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # kubectl 설치 방법 변경\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([ \\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods in all namespaces\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(f\\\\\\\"\\\\u003chtml\\\\u003e\\\\u003cbody\\\\u003e\\\\u003ch1\\\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\\\u003c/h1\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in describe.splitlines() if keyword in l])\\\\n            matched_logs = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in logs.splitlines() if keyword in l])\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003ch2\\\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\\\u003c/h2\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eDescribe\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_describe}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eLogs\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_logs}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in events.splitlines() if keyword in l])\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        # Write filtered events grouped by namespace\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003ch3\\\\u003eEvents in Namespace: `{ns}`\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"'  \\\\u0026\\\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str = \\\\\\\"main\\\\\\\"\\\\n):\\\\n\\\\n    import subprocess\\\\n    import shutil\\\\n    import os\\\\n    from datetime import datetime\\\\n\\\\n    os.environ[\\\\\\\"GIT_ASKPASS\\\\\\\"] = \\\\\\\"echo\\\\\\\"\\\\n\\\\n    report_path = report_file.path\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_path)\\\\n    if not os.path.exists(report_path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_path} does not exist\\\\\\\")\\\\n\\\\n    # HTML 파일 이름 만들기\\\\n    timestamp = datetime.now().strftime(\\\\\\\"report-%Y%m%d-%H%M%S.html\\\\\\\")\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n    report_target_path = os.path.join(repo_dir, timestamp)\\\\n\\\\n    # Git clone\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    shutil.copyfile(report_path, report_target_path)\\\\n\\\\n    # Git config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    # Git commit \\\\u0026 push\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", f\\\\\\\"Add {timestamp}\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(f\\\\\\\"Report committed and pushed to GitHub as {timestamp}.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Clone of Run of pod_log_pipeline_advance_version_at_2025-05-08T05:25:35.947Z (05c63)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T05:41:06Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}},{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T05:41:16Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-filtered-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-filtered-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-filtered-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-filtered-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-filtered-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-filtered-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Running\",\"startedAt\":\"2025-05-08T05:41:06Z\",\"finishedAt\":null,\"progress\":\"1/2\",\"nodes\":{\"pod-log-to-github-zj76w\":{\"id\":\"pod-log-to-github-zj76w\",\"name\":\"pod-log-to-github-zj76w\",\"displayName\":\"pod-log-to-github-zj76w\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-zj76w\",\"phase\":\"Running\",\"startedAt\":\"2025-05-08T05:41:06Z\",\"finishedAt\":null,\"progress\":\"1/2\",\"children\":[\"pod-log-to-github-zj76w-1372256085\"]},\"pod-log-to-github-zj76w-1372256085\":{\"id\":\"pod-log-to-github-zj76w-1372256085\",\"name\":\"pod-log-to-github-zj76w.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-zj76w\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-zj76w\",\"startedAt\":\"2025-05-08T05:41:06Z\",\"finishedAt\":\"2025-05-08T05:41:10Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":3,\"memory\":2},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"value\":\"125\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"value\":\"0\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-zj76w/2025/05/08/pod-log-to-github-zj76w-system-dag-driver-1372256085/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-zj76w-402210888\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-zj76w-402210888\":{\"id\":\"pod-log-to-github-zj76w-402210888\",\"name\":\"pod-log-to-github-zj76w.root\",\"displayName\":\"root\",\"type\":\"DAG\",\"templateName\":\"root\",\"templateScope\":\"local/pod-log-to-github-zj76w\",\"phase\":\"Running\",\"boundaryID\":\"pod-log-to-github-zj76w\",\"startedAt\":\"2025-05-08T05:41:16Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"125\"}]},\"children\":[\"pod-log-to-github-zj76w-878217725\"]},\"pod-log-to-github-zj76w-878217725\":{\"id\":\"pod-log-to-github-zj76w-878217725\",\"name\":\"pod-log-to-github-zj76w.root.collect-filtered-logs-driver\",\"displayName\":\"collect-filtered-logs-driver\",\"type\":\"Pod\",\"templateName\":\"system-container-driver\",\"templateScope\":\"local/pod-log-to-github-zj76w\",\"phase\":\"Pending\",\"boundaryID\":\"pod-log-to-github-zj76w-402210888\",\"startedAt\":\"2025-05-08T05:41:16Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"defaultValue\\\":\\\"crash\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"}}},\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\\\\\"crash\\\\\\\"):\\\\n    import shutil\\\\n    import json\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # kubectl 설치 방법 변경\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([ \\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods in all namespaces\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(f\\\\\\\"\\\\u003chtml\\\\u003e\\\\u003cbody\\\\u003e\\\\u003ch1\\\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\\\u003c/h1\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in describe.splitlines() if keyword in l])\\\\n            matched_logs = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in logs.splitlines() if keyword in l])\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003ch2\\\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\\\u003c/h2\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eDescribe\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_describe}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eLogs\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_logs}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in events.splitlines() if keyword in l])\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        # Write filtered events grouped by namespace\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003ch3\\\\u003eEvents in Namespace: `{ns}`\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\"},{\"name\":\"parent-dag-id\",\"value\":\"125\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\",\"value\":\"\"}]}}},\"conditions\":[{\"type\":\"PodRunning\",\"status\":\"False\"}],\"resourcesDuration\":{\"cpu\":3,\"memory\":2},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = Unauthenticated desc = Jwt is expired"
time="2025-05-08T05:41:51Z" level=error msg="Transient failure while syncing resource (kubeflow-user-example-com/pod-log-to-github-zj76w): CustomError (code: 0): Syncing Workflow (pod-log-to-github-zj76w): transient failure: CustomError (code: 0): Error while reporting workflow resource (code: Unauthenticated, message: Jwt is expired): rpc error: code = Unauthenticated desc = Jwt is expired, {\"metadata\":{\"name\":\"pod-log-to-github-zj76w\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"3066c70d-3732-4a31-830b-af01907bb8cd\",\"resourceVersion\":\"6535565\",\"generation\":5,\"creationTimestamp\":\"2025-05-08T05:41:06Z\",\"labels\":{\"pipeline/runid\":\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"workflows.argoproj.io/phase\":\"Running\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-filtered-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"defaultValue\\\":\\\"crash\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"}}},\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\\\\\"crash\\\\\\\"):\\\\n    import shutil\\\\n    import json\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # kubectl 설치 방법 변경\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([ \\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods in all namespaces\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(f\\\\\\\"\\\\u003chtml\\\\u003e\\\\u003cbody\\\\u003e\\\\u003ch1\\\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\\\u003c/h1\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in describe.splitlines() if keyword in l])\\\\n            matched_logs = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in logs.splitlines() if keyword in l])\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003ch2\\\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\\\u003c/h2\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eDescribe\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_describe}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eLogs\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_logs}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in events.splitlines() if keyword in l])\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        # Write filtered events grouped by namespace\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003ch3\\\\u003eEvents in Namespace: `{ns}`\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"'  \\\\u0026\\\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str = \\\\\\\"main\\\\\\\"\\\\n):\\\\n\\\\n    import subprocess\\\\n    import shutil\\\\n    import os\\\\n    from datetime import datetime\\\\n\\\\n    os.environ[\\\\\\\"GIT_ASKPASS\\\\\\\"] = \\\\\\\"echo\\\\\\\"\\\\n\\\\n    report_path = report_file.path\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_path)\\\\n    if not os.path.exists(report_path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_path} does not exist\\\\\\\")\\\\n\\\\n    # HTML 파일 이름 만들기\\\\n    timestamp = datetime.now().strftime(\\\\\\\"report-%Y%m%d-%H%M%S.html\\\\\\\")\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n    report_target_path = os.path.join(repo_dir, timestamp)\\\\n\\\\n    # Git clone\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    shutil.copyfile(report_path, report_target_path)\\\\n\\\\n    # Git config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    # Git commit \\\\u0026 push\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", f\\\\\\\"Add {timestamp}\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(f\\\\\\\"Report committed and pushed to GitHub as {timestamp}.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Clone of Run of pod_log_pipeline_advance_version_at_2025-05-08T05:25:35.947Z (05c63)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T05:41:06Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}},{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T05:41:36Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-filtered-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-filtered-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-filtered-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-filtered-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-filtered-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-filtered-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Running\",\"startedAt\":\"2025-05-08T05:41:06Z\",\"finishedAt\":null,\"progress\":\"2/3\",\"nodes\":{\"pod-log-to-github-zj76w\":{\"id\":\"pod-log-to-github-zj76w\",\"name\":\"pod-log-to-github-zj76w\",\"displayName\":\"pod-log-to-github-zj76w\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-zj76w\",\"phase\":\"Running\",\"startedAt\":\"2025-05-08T05:41:06Z\",\"finishedAt\":null,\"progress\":\"2/3\",\"children\":[\"pod-log-to-github-zj76w-1372256085\"]},\"pod-log-to-github-zj76w-1372256085\":{\"id\":\"pod-log-to-github-zj76w-1372256085\",\"name\":\"pod-log-to-github-zj76w.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-zj76w\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-zj76w\",\"startedAt\":\"2025-05-08T05:41:06Z\",\"finishedAt\":\"2025-05-08T05:41:10Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":3,\"memory\":2},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"value\":\"125\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"value\":\"0\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-zj76w/2025/05/08/pod-log-to-github-zj76w-system-dag-driver-1372256085/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-zj76w-402210888\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-zj76w-2176256896\":{\"id\":\"pod-log-to-github-zj76w-2176256896\",\"name\":\"pod-log-to-github-zj76w.root.collect-filtered-logs\",\"displayName\":\"collect-filtered-logs\",\"type\":\"DAG\",\"templateName\":\"system-container-executor\",\"templateScope\":\"local/pod-log-to-github-zj76w\",\"phase\":\"Running\",\"boundaryID\":\"pod-log-to-github-zj76w-402210888\",\"startedAt\":\"2025-05-08T05:41:26Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"python:3.9\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\\\",\\\"--execution_id\\\",\\\"126\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/2e7ffed3-456d-4740-98ac-f89e1c2eb2b9/collect-filtered-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"inputDefinitions\\\\\\\":{\\\\\\\"parameters\\\\\\\":{\\\\\\\"keyword\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\", \\\\\\\"defaultValue\\\\\\\":\\\\\\\"crash\\\\\\\", \\\\\\\"isOptional\\\\\\\":true}}}, \\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-filtered-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\\\\\"crash\\\\\\\"):\\\\n    import shutil\\\\n    import json\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # kubectl 설치 방법 변경\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([ \\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods in all namespaces\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(f\\\\\\\"\\\\u003chtml\\\\u003e\\\\u003cbody\\\\u003e\\\\u003ch1\\\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\\\u003c/h1\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in describe.splitlines() if keyword in l])\\\\n            matched_logs = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in logs.splitlines() if keyword in l])\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003ch2\\\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\\\u003c/h2\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eDescribe\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_describe}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eLogs\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_logs}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in events.splitlines() if keyword in l])\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        # Write filtered events grouped by namespace\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003ch3\\\\u003eEvents in Namespace: `{ns}`\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"resources\\\":{}}]}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"false\"}]},\"children\":[\"pod-log-to-github-zj76w-3220014891\"]},\"pod-log-to-github-zj76w-3220014891\":{\"id\":\"pod-log-to-github-zj76w-3220014891\",\"name\":\"pod-log-to-github-zj76w.root.collect-filtered-logs.executor\",\"displayName\":\"executor\",\"type\":\"Pod\",\"templateName\":\"system-container-impl\",\"templateScope\":\"local/pod-log-to-github-zj76w\",\"phase\":\"Running\",\"boundaryID\":\"pod-log-to-github-zj76w-2176256896\",\"startedAt\":\"2025-05-08T05:41:26Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"python:3.9\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\\\",\\\"--execution_id\\\",\\\"126\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/2e7ffed3-456d-4740-98ac-f89e1c2eb2b9/collect-filtered-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"inputDefinitions\\\\\\\":{\\\\\\\"parameters\\\\\\\":{\\\\\\\"keyword\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\", \\\\\\\"defaultValue\\\\\\\":\\\\\\\"crash\\\\\\\", \\\\\\\"isOptional\\\\\\\":true}}}, \\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-filtered-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\\\\\"crash\\\\\\\"):\\\\n    import shutil\\\\n    import json\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # kubectl 설치 방법 변경\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([ \\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods in all namespaces\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(f\\\\\\\"\\\\u003chtml\\\\u003e\\\\u003cbody\\\\u003e\\\\u003ch1\\\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\\\u003c/h1\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in describe.splitlines() if keyword in l])\\\\n            matched_logs = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in logs.splitlines() if keyword in l])\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003ch2\\\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\\\u003c/h2\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eDescribe\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_describe}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eLogs\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_logs}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in events.splitlines() if keyword in l])\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        # Write filtered events grouped by namespace\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003ch3\\\\u003eEvents in Namespace: `{ns}`\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"resources\\\":{}}]}\"}]},\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-zj76w-402210888\":{\"id\":\"pod-log-to-github-zj76w-402210888\",\"name\":\"pod-log-to-github-zj76w.root\",\"displayName\":\"root\",\"type\":\"DAG\",\"templateName\":\"root\",\"templateScope\":\"local/pod-log-to-github-zj76w\",\"phase\":\"Running\",\"boundaryID\":\"pod-log-to-github-zj76w\",\"startedAt\":\"2025-05-08T05:41:16Z\",\"finishedAt\":null,\"progress\":\"1/2\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"125\"}]},\"children\":[\"pod-log-to-github-zj76w-878217725\"]},\"pod-log-to-github-zj76w-878217725\":{\"id\":\"pod-log-to-github-zj76w-878217725\",\"name\":\"pod-log-to-github-zj76w.root.collect-filtered-logs-driver\",\"displayName\":\"collect-filtered-logs-driver\",\"type\":\"Pod\",\"templateName\":\"system-container-driver\",\"templateScope\":\"local/pod-log-to-github-zj76w\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-zj76w-402210888\",\"startedAt\":\"2025-05-08T05:41:16Z\",\"finishedAt\":\"2025-05-08T05:41:20Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":3,\"memory\":2},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"defaultValue\\\":\\\"crash\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"}}},\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\\\\\"crash\\\\\\\"):\\\\n    import shutil\\\\n    import json\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # kubectl 설치 방법 변경\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([ \\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods in all namespaces\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(f\\\\\\\"\\\\u003chtml\\\\u003e\\\\u003cbody\\\\u003e\\\\u003ch1\\\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\\\u003c/h1\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in describe.splitlines() if keyword in l])\\\\n            matched_logs = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in logs.splitlines() if keyword in l])\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003ch2\\\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\\\u003c/h2\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eDescribe\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_describe}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eLogs\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_logs}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in events.splitlines() if keyword in l])\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        # Write filtered events grouped by namespace\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003ch3\\\\u003eEvents in Namespace: `{ns}`\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\"},{\"name\":\"parent-dag-id\",\"value\":\"125\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\",\"value\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"python:3.9\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\\\",\\\"--execution_id\\\",\\\"126\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/2e7ffed3-456d-4740-98ac-f89e1c2eb2b9/collect-filtered-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"inputDefinitions\\\\\\\":{\\\\\\\"parameters\\\\\\\":{\\\\\\\"keyword\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\", \\\\\\\"defaultValue\\\\\\\":\\\\\\\"crash\\\\\\\", \\\\\\\"isOptional\\\\\\\":true}}}, \\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-filtered-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\\\\\"crash\\\\\\\"):\\\\n    import shutil\\\\n    import json\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # kubectl 설치 방법 변경\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([ \\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods in all namespaces\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(f\\\\\\\"\\\\u003chtml\\\\u003e\\\\u003cbody\\\\u003e\\\\u003ch1\\\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\\\u003c/h1\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in describe.splitlines() if keyword in l])\\\\n            matched_logs = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in logs.splitlines() if keyword in l])\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003ch2\\\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\\\u003c/h2\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eDescribe\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_describe}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eLogs\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_logs}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in events.splitlines() if keyword in l])\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        # Write filtered events grouped by namespace\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003ch3\\\\u003eEvents in Namespace: `{ns}`\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"resources\\\":{}}]}\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-zj76w/2025/05/08/pod-log-to-github-zj76w-system-container-driver-878217725/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-zj76w-2176256896\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"}},\"conditions\":[{\"type\":\"PodRunning\",\"status\":\"True\"}],\"resourcesDuration\":{\"cpu\":6,\"memory\":4},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = Unauthenticated desc = Jwt is expired: CustomError (code: 0): Error while reporting workflow resource (code: Unauthenticated, message: Jwt is expired): rpc error: code = Unauthenticated desc = Jwt is expired, {\"metadata\":{\"name\":\"pod-log-to-github-zj76w\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"3066c70d-3732-4a31-830b-af01907bb8cd\",\"resourceVersion\":\"6535565\",\"generation\":5,\"creationTimestamp\":\"2025-05-08T05:41:06Z\",\"labels\":{\"pipeline/runid\":\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"workflows.argoproj.io/phase\":\"Running\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-filtered-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"defaultValue\\\":\\\"crash\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"}}},\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\\\\\"crash\\\\\\\"):\\\\n    import shutil\\\\n    import json\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # kubectl 설치 방법 변경\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([ \\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods in all namespaces\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(f\\\\\\\"\\\\u003chtml\\\\u003e\\\\u003cbody\\\\u003e\\\\u003ch1\\\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\\\u003c/h1\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in describe.splitlines() if keyword in l])\\\\n            matched_logs = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in logs.splitlines() if keyword in l])\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003ch2\\\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\\\u003c/h2\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eDescribe\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_describe}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eLogs\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_logs}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in events.splitlines() if keyword in l])\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        # Write filtered events grouped by namespace\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003ch3\\\\u003eEvents in Namespace: `{ns}`\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"'  \\\\u0026\\\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str = \\\\\\\"main\\\\\\\"\\\\n):\\\\n\\\\n    import subprocess\\\\n    import shutil\\\\n    import os\\\\n    from datetime import datetime\\\\n\\\\n    os.environ[\\\\\\\"GIT_ASKPASS\\\\\\\"] = \\\\\\\"echo\\\\\\\"\\\\n\\\\n    report_path = report_file.path\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_path)\\\\n    if not os.path.exists(report_path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_path} does not exist\\\\\\\")\\\\n\\\\n    # HTML 파일 이름 만들기\\\\n    timestamp = datetime.now().strftime(\\\\\\\"report-%Y%m%d-%H%M%S.html\\\\\\\")\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n    report_target_path = os.path.join(repo_dir, timestamp)\\\\n\\\\n    # Git clone\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    shutil.copyfile(report_path, report_target_path)\\\\n\\\\n    # Git config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    # Git commit \\\\u0026 push\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", f\\\\\\\"Add {timestamp}\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(f\\\\\\\"Report committed and pushed to GitHub as {timestamp}.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Clone of Run of pod_log_pipeline_advance_version_at_2025-05-08T05:25:35.947Z (05c63)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T05:41:06Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}},{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T05:41:36Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-filtered-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-filtered-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-filtered-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-filtered-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-filtered-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-filtered-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Running\",\"startedAt\":\"2025-05-08T05:41:06Z\",\"finishedAt\":null,\"progress\":\"2/3\",\"nodes\":{\"pod-log-to-github-zj76w\":{\"id\":\"pod-log-to-github-zj76w\",\"name\":\"pod-log-to-github-zj76w\",\"displayName\":\"pod-log-to-github-zj76w\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-zj76w\",\"phase\":\"Running\",\"startedAt\":\"2025-05-08T05:41:06Z\",\"finishedAt\":null,\"progress\":\"2/3\",\"children\":[\"pod-log-to-github-zj76w-1372256085\"]},\"pod-log-to-github-zj76w-1372256085\":{\"id\":\"pod-log-to-github-zj76w-1372256085\",\"name\":\"pod-log-to-github-zj76w.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-zj76w\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-zj76w\",\"startedAt\":\"2025-05-08T05:41:06Z\",\"finishedAt\":\"2025-05-08T05:41:10Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":3,\"memory\":2},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"value\":\"125\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"value\":\"0\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-zj76w/2025/05/08/pod-log-to-github-zj76w-system-dag-driver-1372256085/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-zj76w-402210888\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-zj76w-2176256896\":{\"id\":\"pod-log-to-github-zj76w-2176256896\",\"name\":\"pod-log-to-github-zj76w.root.collect-filtered-logs\",\"displayName\":\"collect-filtered-logs\",\"type\":\"DAG\",\"templateName\":\"system-container-executor\",\"templateScope\":\"local/pod-log-to-github-zj76w\",\"phase\":\"Running\",\"boundaryID\":\"pod-log-to-github-zj76w-402210888\",\"startedAt\":\"2025-05-08T05:41:26Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"python:3.9\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\\\",\\\"--execution_id\\\",\\\"126\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/2e7ffed3-456d-4740-98ac-f89e1c2eb2b9/collect-filtered-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"inputDefinitions\\\\\\\":{\\\\\\\"parameters\\\\\\\":{\\\\\\\"keyword\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\", \\\\\\\"defaultValue\\\\\\\":\\\\\\\"crash\\\\\\\", \\\\\\\"isOptional\\\\\\\":true}}}, \\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-filtered-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\\\\\"crash\\\\\\\"):\\\\n    import shutil\\\\n    import json\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # kubectl 설치 방법 변경\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([ \\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods in all namespaces\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(f\\\\\\\"\\\\u003chtml\\\\u003e\\\\u003cbody\\\\u003e\\\\u003ch1\\\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\\\u003c/h1\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in describe.splitlines() if keyword in l])\\\\n            matched_logs = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in logs.splitlines() if keyword in l])\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003ch2\\\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\\\u003c/h2\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eDescribe\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_describe}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eLogs\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_logs}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in events.splitlines() if keyword in l])\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        # Write filtered events grouped by namespace\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003ch3\\\\u003eEvents in Namespace: `{ns}`\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"resources\\\":{}}]}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"false\"}]},\"children\":[\"pod-log-to-github-zj76w-3220014891\"]},\"pod-log-to-github-zj76w-3220014891\":{\"id\":\"pod-log-to-github-zj76w-3220014891\",\"name\":\"pod-log-to-github-zj76w.root.collect-filtered-logs.executor\",\"displayName\":\"executor\",\"type\":\"Pod\",\"templateName\":\"system-container-impl\",\"templateScope\":\"local/pod-log-to-github-zj76w\",\"phase\":\"Running\",\"boundaryID\":\"pod-log-to-github-zj76w-2176256896\",\"startedAt\":\"2025-05-08T05:41:26Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"python:3.9\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\\\",\\\"--execution_id\\\",\\\"126\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/2e7ffed3-456d-4740-98ac-f89e1c2eb2b9/collect-filtered-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"inputDefinitions\\\\\\\":{\\\\\\\"parameters\\\\\\\":{\\\\\\\"keyword\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\", \\\\\\\"defaultValue\\\\\\\":\\\\\\\"crash\\\\\\\", \\\\\\\"isOptional\\\\\\\":true}}}, \\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-filtered-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\\\\\"crash\\\\\\\"):\\\\n    import shutil\\\\n    import json\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # kubectl 설치 방법 변경\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([ \\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods in all namespaces\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(f\\\\\\\"\\\\u003chtml\\\\u003e\\\\u003cbody\\\\u003e\\\\u003ch1\\\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\\\u003c/h1\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in describe.splitlines() if keyword in l])\\\\n            matched_logs = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in logs.splitlines() if keyword in l])\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003ch2\\\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\\\u003c/h2\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eDescribe\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_describe}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eLogs\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_logs}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in events.splitlines() if keyword in l])\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        # Write filtered events grouped by namespace\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003ch3\\\\u003eEvents in Namespace: `{ns}`\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"resources\\\":{}}]}\"}]},\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-zj76w-402210888\":{\"id\":\"pod-log-to-github-zj76w-402210888\",\"name\":\"pod-log-to-github-zj76w.root\",\"displayName\":\"root\",\"type\":\"DAG\",\"templateName\":\"root\",\"templateScope\":\"local/pod-log-to-github-zj76w\",\"phase\":\"Running\",\"boundaryID\":\"pod-log-to-github-zj76w\",\"startedAt\":\"2025-05-08T05:41:16Z\",\"finishedAt\":null,\"progress\":\"1/2\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"125\"}]},\"children\":[\"pod-log-to-github-zj76w-878217725\"]},\"pod-log-to-github-zj76w-878217725\":{\"id\":\"pod-log-to-github-zj76w-878217725\",\"name\":\"pod-log-to-github-zj76w.root.collect-filtered-logs-driver\",\"displayName\":\"collect-filtered-logs-driver\",\"type\":\"Pod\",\"templateName\":\"system-container-driver\",\"templateScope\":\"local/pod-log-to-github-zj76w\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-zj76w-402210888\",\"startedAt\":\"2025-05-08T05:41:16Z\",\"finishedAt\":\"2025-05-08T05:41:20Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":3,\"memory\":2},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"defaultValue\\\":\\\"crash\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"}}},\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\\\\\"crash\\\\\\\"):\\\\n    import shutil\\\\n    import json\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # kubectl 설치 방법 변경\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([ \\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods in all namespaces\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(f\\\\\\\"\\\\u003chtml\\\\u003e\\\\u003cbody\\\\u003e\\\\u003ch1\\\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\\\u003c/h1\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in describe.splitlines() if keyword in l])\\\\n            matched_logs = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in logs.splitlines() if keyword in l])\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003ch2\\\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\\\u003c/h2\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eDescribe\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_describe}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eLogs\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_logs}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in events.splitlines() if keyword in l])\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        # Write filtered events grouped by namespace\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003ch3\\\\u003eEvents in Namespace: `{ns}`\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\"},{\"name\":\"parent-dag-id\",\"value\":\"125\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\",\"value\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"python:3.9\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"2e7ffed3-456d-4740-98ac-f89e1c2eb2b9\\\",\\\"--execution_id\\\",\\\"126\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/2e7ffed3-456d-4740-98ac-f89e1c2eb2b9/collect-filtered-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"inputDefinitions\\\\\\\":{\\\\\\\"parameters\\\\\\\":{\\\\\\\"keyword\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\", \\\\\\\"defaultValue\\\\\\\":\\\\\\\"crash\\\\\\\", \\\\\\\"isOptional\\\\\\\":true}}}, \\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-filtered-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\\\\\"crash\\\\\\\"):\\\\n    import shutil\\\\n    import json\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # kubectl 설치 방법 변경\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([ \\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods in all namespaces\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(f\\\\\\\"\\\\u003chtml\\\\u003e\\\\u003cbody\\\\u003e\\\\u003ch1\\\\u003e장애 복기 리포트 (keyword: `{keyword}`)\\\\u003c/h1\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in describe.splitlines() if keyword in l])\\\\n            matched_logs = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in logs.splitlines() if keyword in l])\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003ch2\\\\u003ePod: `{pod_name}` (Namespace: `{namespace}`)\\\\u003c/h2\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eDescribe\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_describe}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(f\\\\\\\"\\\\u003ch3\\\\u003eLogs\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{matched_logs}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in events.splitlines() if keyword in l])\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        # Write filtered events grouped by namespace\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003ch3\\\\u003eEvents in Namespace: `{ns}`\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\\\\\n\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"resources\\\":{}}]}\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-zj76w/2025/05/08/pod-log-to-github-zj76w-system-container-driver-878217725/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-zj76w-2176256896\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"}},\"conditions\":[{\"type\":\"PodRunning\",\"status\":\"True\"}],\"resourcesDuration\":{\"cpu\":6,\"memory\":4},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = Unauthenticated desc = Jwt is expired"
time="2025-05-08T05:44:53Z" level=error msg="Permanent failure while syncing resource (kubeflow-user-example-com/pod-log-to-github-qv4f5): CustomError (code: 1): Syncing Workflow (pod-log-to-github-qv4f5): permanent failure: CustomError (code: 1): Error while reporting workflow resource (code: NotFound, message: Failed to report workflow: NotFoundError: Failed to add PersistedFinalState label to workflow pod-log-to-github-qv4f5: workflows.argoproj.io \"pod-log-to-github-qv4f5\" not found): rpc error: code = NotFound desc = Failed to report workflow: NotFoundError: Failed to add PersistedFinalState label to workflow pod-log-to-github-qv4f5: workflows.argoproj.io \"pod-log-to-github-qv4f5\" not found, {\"metadata\":{\"name\":\"pod-log-to-github-qv4f5\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"239fd9c0-d345-430a-bfd0-9980c44a949b\",\"resourceVersion\":\"6049335\",\"generation\":6,\"creationTimestamp\":\"2025-05-07T05:43:53Z\",\"labels\":{\"pipeline/persistedFinalState\":\"true\",\"pipeline/runid\":\"e217d06b-0721-4a9b-99c2-bc2124e299ff\",\"workflows.argoproj.io/completed\":\"true\",\"workflows.argoproj.io/phase\":\"Failed\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-pod-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-pod-logs\\\",\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"pod_name\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}},\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-pod-logs\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-pod-logs\\\"},\\\"inputs\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"componentInputParameter\\\":\\\"namespace\\\"},\\\"pod_name\\\":{\\\"componentInputParameter\\\":\\\"pod_name\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-pod-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-pod-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-pod-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"namespace\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"pod_name\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-pod-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_pod_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_pod_logs(\\\\n    namespace: str,\\\\n    pod_name: str,\\\\n    output_report: Output[Dataset]\\\\n):\\\\n    import subprocess\\\\n\\\\n    describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n    logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n    events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n\\\\n    with open(output_report.path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(\\\\\\\"# 장애 복기 리포트\\\\\\\\n\\\\\\\")\\\\n        f.write(f\\\\\\\"**Pod**: {pod_name}\\\\\\\\n\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## describe\\\\\\\\n```text\\\\\\\\n\\\\\\\" + describe + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## logs\\\\\\\\n```text\\\\\\\\n\\\\\\\" + logs + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## events\\\\\\\\n```text\\\\\\\\n\\\\\\\" + events + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"bitnami/kubectl:1.26\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str = \\\\\\\"main\\\\\\\"\\\\n):\\\\n    import os\\\\n    import subprocess\\\\n\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_file.path)\\\\n    if not os.path.exists(report_file.path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_file.path} does not exist\\\\\\\")\\\\n\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", report_path, os.path.join(repo_dir, \\\\\\\"report.md\\\\\\\")], check=True)\\\\n\\\\n    # Git user config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", \\\\\\\"Add report\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(\\\\\\\"Report committed and pushed to GitHub.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Run of pod_log_pipeline_version_at_2025-05-07T05:43:33.961Z (d18a3)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-07T05:44:47Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/completed\":{},\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}},{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-07T05:50:38Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-pod-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-pod-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/persistedFinalState\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"e217d06b-0721-4a9b-99c2-bc2124e299ff\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-pod-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-pod-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-pod-logs\\\"},\\\"inputs\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"componentInputParameter\\\":\\\"namespace\\\"},\\\"pod_name\\\":{\\\"componentInputParameter\\\":\\\"pod_name\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-pod-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-pod-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-pod-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-pod-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-pod-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-pod-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-pod-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-pod-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-pod-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"e217d06b-0721-4a9b-99c2-bc2124e299ff\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\",\\\"github_token\\\":\\\"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\\\",\\\"namespace\\\":\\\"istio-system\\\",\\\"pod_name\\\":\\\"toolbox\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"e217d06b-0721-4a9b-99c2-bc2124e299ff\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Failed\",\"startedAt\":\"2025-05-07T05:43:53Z\",\"finishedAt\":\"2025-05-07T05:44:47Z\",\"progress\":\"2/3\",\"nodes\":{\"pod-log-to-github-qv4f5\":{\"id\":\"pod-log-to-github-qv4f5\",\"name\":\"pod-log-to-github-qv4f5\",\"displayName\":\"pod-log-to-github-qv4f5\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-qv4f5\",\"phase\":\"Failed\",\"startedAt\":\"2025-05-07T05:43:53Z\",\"finishedAt\":\"2025-05-07T05:44:47Z\",\"progress\":\"2/3\",\"resourcesDuration\":{\"cpu\":35,\"memory\":33},\"children\":[\"pod-log-to-github-qv4f5-3776246499\"],\"outboundNodes\":[\"pod-log-to-github-qv4f5-1172358263\"]},\"pod-log-to-github-qv4f5-1172358263\":{\"id\":\"pod-log-to-github-qv4f5-1172358263\",\"name\":\"pod-log-to-github-qv4f5.root.upload-to-github\",\"displayName\":\"upload-to-github\",\"type\":\"Skipped\",\"templateName\":\"system-container-executor\",\"templateScope\":\"local/pod-log-to-github-qv4f5\",\"phase\":\"Omitted\",\"boundaryID\":\"pod-log-to-github-qv4f5-2214199526\",\"message\":\"omitted: depends condition not met\",\"startedAt\":\"2025-05-07T05:44:47Z\",\"finishedAt\":\"2025-05-07T05:44:47Z\"},\"pod-log-to-github-qv4f5-1246834064\":{\"id\":\"pod-log-to-github-qv4f5-1246834064\",\"name\":\"pod-log-to-github-qv4f5.root.upload-to-github-driver\",\"displayName\":\"upload-to-github-driver\",\"type\":\"Skipped\",\"templateName\":\"system-container-driver\",\"templateScope\":\"local/pod-log-to-github-qv4f5\",\"phase\":\"Omitted\",\"boundaryID\":\"pod-log-to-github-qv4f5-2214199526\",\"message\":\"omitted: depends condition not met\",\"startedAt\":\"2025-05-07T05:44:47Z\",\"finishedAt\":\"2025-05-07T05:44:47Z\",\"children\":[\"pod-log-to-github-qv4f5-1172358263\"]},\"pod-log-to-github-qv4f5-2146131056\":{\"id\":\"pod-log-to-github-qv4f5-2146131056\",\"name\":\"pod-log-to-github-qv4f5.root.collect-pod-logs\",\"displayName\":\"collect-pod-logs\",\"type\":\"DAG\",\"templateName\":\"system-container-executor\",\"templateScope\":\"local/pod-log-to-github-qv4f5\",\"phase\":\"Failed\",\"boundaryID\":\"pod-log-to-github-qv4f5-2214199526\",\"startedAt\":\"2025-05-07T05:44:13Z\",\"finishedAt\":\"2025-05-07T05:44:47Z\",\"progress\":\"0/1\",\"resourcesDuration\":{\"cpu\":26,\"memory\":26},\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"bitnami/kubectl:1.26\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"e217d06b-0721-4a9b-99c2-bc2124e299ff\\\",\\\"--execution_id\\\",\\\"34\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{\\\\\\\"parameterValues\\\\\\\":{\\\\\\\"namespace\\\\\\\":\\\\\\\"istio-system\\\\\\\", \\\\\\\"pod_name\\\\\\\":\\\\\\\"toolbox\\\\\\\"}}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/e217d06b-0721-4a9b-99c2-bc2124e299ff/collect-pod-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"inputDefinitions\\\\\\\":{\\\\\\\"parameters\\\\\\\":{\\\\\\\"namespace\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}, \\\\\\\"pod_name\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}}}, \\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-pod-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_pod_logs(\\\\n    namespace: str,\\\\n    pod_name: str,\\\\n    output_report: Output[Dataset]\\\\n):\\\\n    import subprocess\\\\n\\\\n    describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n    logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n    events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n\\\\n    with open(output_report.path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(\\\\\\\"# 장애 복기 리포트\\\\\\\\n\\\\\\\")\\\\n        f.write(f\\\\\\\"**Pod**: {pod_name}\\\\\\\\n\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## describe\\\\\\\\n```text\\\\\\\\n\\\\\\\" + describe + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## logs\\\\\\\\n```text\\\\\\\\n\\\\\\\" + logs + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## events\\\\\\\\n```text\\\\\\\\n\\\\\\\" + events + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_pod_logs\\\"],\\\"resources\\\":{}}]}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"false\"}]},\"children\":[\"pod-log-to-github-qv4f5-742416059\"],\"outboundNodes\":[\"pod-log-to-github-qv4f5-742416059\"]},\"pod-log-to-github-qv4f5-2214199526\":{\"id\":\"pod-log-to-github-qv4f5-2214199526\",\"name\":\"pod-log-to-github-qv4f5.root\",\"displayName\":\"root\",\"type\":\"DAG\",\"templateName\":\"root\",\"templateScope\":\"local/pod-log-to-github-qv4f5\",\"phase\":\"Failed\",\"boundaryID\":\"pod-log-to-github-qv4f5\",\"startedAt\":\"2025-05-07T05:44:03Z\",\"finishedAt\":\"2025-05-07T05:44:47Z\",\"progress\":\"1/2\",\"resourcesDuration\":{\"cpu\":31,\"memory\":30},\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"33\"}]},\"children\":[\"pod-log-to-github-qv4f5-4185791085\"],\"outboundNodes\":[\"pod-log-to-github-qv4f5-1172358263\"]},\"pod-log-to-github-qv4f5-3776246499\":{\"id\":\"pod-log-to-github-qv4f5-3776246499\",\"name\":\"pod-log-to-github-qv4f5.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-qv4f5\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-qv4f5\",\"startedAt\":\"2025-05-07T05:43:53Z\",\"finishedAt\":\"2025-05-07T05:43:56Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":4,\"memory\":3},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-pod-logs\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-pod-logs\\\"},\\\"inputs\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"componentInputParameter\\\":\\\"namespace\\\"},\\\"pod_name\\\":{\\\"componentInputParameter\\\":\\\"pod_name\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-pod-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-pod-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-pod-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"namespace\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"pod_name\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\",\\\"github_token\\\":\\\"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\\\",\\\"namespace\\\":\\\"istio-system\\\",\\\"pod_name\\\":\\\"toolbox\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"value\":\"33\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"value\":\"0\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-qv4f5/2025/05/07/pod-log-to-github-qv4f5-system-dag-driver-3776246499/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-qv4f5-2214199526\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-qv4f5-4185791085\":{\"id\":\"pod-log-to-github-qv4f5-4185791085\",\"name\":\"pod-log-to-github-qv4f5.root.collect-pod-logs-driver\",\"displayName\":\"collect-pod-logs-driver\",\"type\":\"Pod\",\"templateName\":\"system-container-driver\",\"templateScope\":\"local/pod-log-to-github-qv4f5\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-qv4f5-2214199526\",\"startedAt\":\"2025-05-07T05:44:03Z\",\"finishedAt\":\"2025-05-07T05:44:06Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":5,\"memory\":4},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"executorLabel\\\":\\\"exec-collect-pod-logs\\\",\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"pod_name\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}},\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-pod-logs\\\"},\\\"inputs\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"componentInputParameter\\\":\\\"namespace\\\"},\\\"pod_name\\\":{\\\"componentInputParameter\\\":\\\"pod_name\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-pod-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_pod_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_pod_logs(\\\\n    namespace: str,\\\\n    pod_name: str,\\\\n    output_report: Output[Dataset]\\\\n):\\\\n    import subprocess\\\\n\\\\n    describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n    logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n    events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n\\\\n    with open(output_report.path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(\\\\\\\"# 장애 복기 리포트\\\\\\\\n\\\\\\\")\\\\n        f.write(f\\\\\\\"**Pod**: {pod_name}\\\\\\\\n\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## describe\\\\\\\\n```text\\\\\\\\n\\\\\\\" + describe + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## logs\\\\\\\\n```text\\\\\\\\n\\\\\\\" + logs + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## events\\\\\\\\n```text\\\\\\\\n\\\\\\\" + events + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"bitnami/kubectl:1.26\\\"}\"},{\"name\":\"parent-dag-id\",\"value\":\"33\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\",\"value\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"bitnami/kubectl:1.26\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"e217d06b-0721-4a9b-99c2-bc2124e299ff\\\",\\\"--execution_id\\\",\\\"34\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{\\\\\\\"parameterValues\\\\\\\":{\\\\\\\"namespace\\\\\\\":\\\\\\\"istio-system\\\\\\\", \\\\\\\"pod_name\\\\\\\":\\\\\\\"toolbox\\\\\\\"}}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/e217d06b-0721-4a9b-99c2-bc2124e299ff/collect-pod-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"inputDefinitions\\\\\\\":{\\\\\\\"parameters\\\\\\\":{\\\\\\\"namespace\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}, \\\\\\\"pod_name\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}}}, \\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-pod-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_pod_logs(\\\\n    namespace: str,\\\\n    pod_name: str,\\\\n    output_report: Output[Dataset]\\\\n):\\\\n    import subprocess\\\\n\\\\n    describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n    logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n    events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n\\\\n    with open(output_report.path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(\\\\\\\"# 장애 복기 리포트\\\\\\\\n\\\\\\\")\\\\n        f.write(f\\\\\\\"**Pod**: {pod_name}\\\\\\\\n\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## describe\\\\\\\\n```text\\\\\\\\n\\\\\\\" + describe + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## logs\\\\\\\\n```text\\\\\\\\n\\\\\\\" + logs + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## events\\\\\\\\n```text\\\\\\\\n\\\\\\\" + events + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_pod_logs\\\"],\\\"resources\\\":{}}]}\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-qv4f5/2025/05/07/pod-log-to-github-qv4f5-system-container-driver-4185791085/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-qv4f5-2146131056\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-qv4f5-742416059\":{\"id\":\"pod-log-to-github-qv4f5-742416059\",\"name\":\"pod-log-to-github-qv4f5.root.collect-pod-logs.executor\",\"displayName\":\"executor\",\"type\":\"Pod\",\"templateName\":\"system-container-impl\",\"templateScope\":\"local/pod-log-to-github-qv4f5\",\"phase\":\"Failed\",\"boundaryID\":\"pod-log-to-github-qv4f5-2146131056\",\"message\":\"Error (exit code 1)\",\"startedAt\":\"2025-05-07T05:44:13Z\",\"finishedAt\":\"2025-05-07T05:44:39Z\",\"progress\":\"0/1\",\"resourcesDuration\":{\"cpu\":26,\"memory\":26},\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"bitnami/kubectl:1.26\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"e217d06b-0721-4a9b-99c2-bc2124e299ff\\\",\\\"--execution_id\\\",\\\"34\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{\\\\\\\"parameterValues\\\\\\\":{\\\\\\\"namespace\\\\\\\":\\\\\\\"istio-system\\\\\\\", \\\\\\\"pod_name\\\\\\\":\\\\\\\"toolbox\\\\\\\"}}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/e217d06b-0721-4a9b-99c2-bc2124e299ff/collect-pod-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"inputDefinitions\\\\\\\":{\\\\\\\"parameters\\\\\\\":{\\\\\\\"namespace\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}, \\\\\\\"pod_name\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}}}, \\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-pod-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_pod_logs(\\\\n    namespace: str,\\\\n    pod_name: str,\\\\n    output_report: Output[Dataset]\\\\n):\\\\n    import subprocess\\\\n\\\\n    describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n    logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n    events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n\\\\n    with open(output_report.path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(\\\\\\\"# 장애 복기 리포트\\\\\\\\n\\\\\\\")\\\\n        f.write(f\\\\\\\"**Pod**: {pod_name}\\\\\\\\n\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## describe\\\\\\\\n```text\\\\\\\\n\\\\\\\" + describe + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## logs\\\\\\\\n```text\\\\\\\\n\\\\\\\" + logs + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## events\\\\\\\\n```text\\\\\\\\n\\\\\\\" + events + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_pod_logs\\\"],\\\"resources\\\":{}}]}\"}]},\"outputs\":{\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-qv4f5/2025/05/07/pod-log-to-github-qv4f5-system-container-impl-742416059/main.log\"}}],\"exitCode\":\"1\"},\"children\":[\"pod-log-to-github-qv4f5-1246834064\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"}},\"conditions\":[{\"type\":\"PodRunning\",\"status\":\"False\"},{\"type\":\"Completed\",\"status\":\"True\"}],\"resourcesDuration\":{\"cpu\":35,\"memory\":33},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = NotFound desc = Failed to report workflow: NotFoundError: Failed to add PersistedFinalState label to workflow pod-log-to-github-qv4f5: workflows.argoproj.io \"pod-log-to-github-qv4f5\" not found: CustomError (code: 1): Error while reporting workflow resource (code: NotFound, message: Failed to report workflow: NotFoundError: Failed to add PersistedFinalState label to workflow pod-log-to-github-qv4f5: workflows.argoproj.io \"pod-log-to-github-qv4f5\" not found): rpc error: code = NotFound desc = Failed to report workflow: NotFoundError: Failed to add PersistedFinalState label to workflow pod-log-to-github-qv4f5: workflows.argoproj.io \"pod-log-to-github-qv4f5\" not found, {\"metadata\":{\"name\":\"pod-log-to-github-qv4f5\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"239fd9c0-d345-430a-bfd0-9980c44a949b\",\"resourceVersion\":\"6049335\",\"generation\":6,\"creationTimestamp\":\"2025-05-07T05:43:53Z\",\"labels\":{\"pipeline/persistedFinalState\":\"true\",\"pipeline/runid\":\"e217d06b-0721-4a9b-99c2-bc2124e299ff\",\"workflows.argoproj.io/completed\":\"true\",\"workflows.argoproj.io/phase\":\"Failed\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-pod-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-pod-logs\\\",\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"pod_name\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}},\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-pod-logs\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-pod-logs\\\"},\\\"inputs\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"componentInputParameter\\\":\\\"namespace\\\"},\\\"pod_name\\\":{\\\"componentInputParameter\\\":\\\"pod_name\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-pod-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-pod-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-pod-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"namespace\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"pod_name\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-pod-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_pod_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_pod_logs(\\\\n    namespace: str,\\\\n    pod_name: str,\\\\n    output_report: Output[Dataset]\\\\n):\\\\n    import subprocess\\\\n\\\\n    describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n    logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n    events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n\\\\n    with open(output_report.path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(\\\\\\\"# 장애 복기 리포트\\\\\\\\n\\\\\\\")\\\\n        f.write(f\\\\\\\"**Pod**: {pod_name}\\\\\\\\n\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## describe\\\\\\\\n```text\\\\\\\\n\\\\\\\" + describe + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## logs\\\\\\\\n```text\\\\\\\\n\\\\\\\" + logs + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## events\\\\\\\\n```text\\\\\\\\n\\\\\\\" + events + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"bitnami/kubectl:1.26\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str = \\\\\\\"main\\\\\\\"\\\\n):\\\\n    import os\\\\n    import subprocess\\\\n\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_file.path)\\\\n    if not os.path.exists(report_file.path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_file.path} does not exist\\\\\\\")\\\\n\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", report_path, os.path.join(repo_dir, \\\\\\\"report.md\\\\\\\")], check=True)\\\\n\\\\n    # Git user config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", \\\\\\\"Add report\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(\\\\\\\"Report committed and pushed to GitHub.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Run of pod_log_pipeline_version_at_2025-05-07T05:43:33.961Z (d18a3)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-07T05:44:47Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/completed\":{},\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}},{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-07T05:50:38Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-pod-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-pod-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/persistedFinalState\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"e217d06b-0721-4a9b-99c2-bc2124e299ff\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-pod-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-pod-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-pod-logs\\\"},\\\"inputs\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"componentInputParameter\\\":\\\"namespace\\\"},\\\"pod_name\\\":{\\\"componentInputParameter\\\":\\\"pod_name\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-pod-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-pod-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-pod-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-pod-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-pod-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-pod-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-pod-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-pod-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-pod-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"e217d06b-0721-4a9b-99c2-bc2124e299ff\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\",\\\"github_token\\\":\\\"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\\\",\\\"namespace\\\":\\\"istio-system\\\",\\\"pod_name\\\":\\\"toolbox\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"e217d06b-0721-4a9b-99c2-bc2124e299ff\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Failed\",\"startedAt\":\"2025-05-07T05:43:53Z\",\"finishedAt\":\"2025-05-07T05:44:47Z\",\"progress\":\"2/3\",\"nodes\":{\"pod-log-to-github-qv4f5\":{\"id\":\"pod-log-to-github-qv4f5\",\"name\":\"pod-log-to-github-qv4f5\",\"displayName\":\"pod-log-to-github-qv4f5\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-qv4f5\",\"phase\":\"Failed\",\"startedAt\":\"2025-05-07T05:43:53Z\",\"finishedAt\":\"2025-05-07T05:44:47Z\",\"progress\":\"2/3\",\"resourcesDuration\":{\"cpu\":35,\"memory\":33},\"children\":[\"pod-log-to-github-qv4f5-3776246499\"],\"outboundNodes\":[\"pod-log-to-github-qv4f5-1172358263\"]},\"pod-log-to-github-qv4f5-1172358263\":{\"id\":\"pod-log-to-github-qv4f5-1172358263\",\"name\":\"pod-log-to-github-qv4f5.root.upload-to-github\",\"displayName\":\"upload-to-github\",\"type\":\"Skipped\",\"templateName\":\"system-container-executor\",\"templateScope\":\"local/pod-log-to-github-qv4f5\",\"phase\":\"Omitted\",\"boundaryID\":\"pod-log-to-github-qv4f5-2214199526\",\"message\":\"omitted: depends condition not met\",\"startedAt\":\"2025-05-07T05:44:47Z\",\"finishedAt\":\"2025-05-07T05:44:47Z\"},\"pod-log-to-github-qv4f5-1246834064\":{\"id\":\"pod-log-to-github-qv4f5-1246834064\",\"name\":\"pod-log-to-github-qv4f5.root.upload-to-github-driver\",\"displayName\":\"upload-to-github-driver\",\"type\":\"Skipped\",\"templateName\":\"system-container-driver\",\"templateScope\":\"local/pod-log-to-github-qv4f5\",\"phase\":\"Omitted\",\"boundaryID\":\"pod-log-to-github-qv4f5-2214199526\",\"message\":\"omitted: depends condition not met\",\"startedAt\":\"2025-05-07T05:44:47Z\",\"finishedAt\":\"2025-05-07T05:44:47Z\",\"children\":[\"pod-log-to-github-qv4f5-1172358263\"]},\"pod-log-to-github-qv4f5-2146131056\":{\"id\":\"pod-log-to-github-qv4f5-2146131056\",\"name\":\"pod-log-to-github-qv4f5.root.collect-pod-logs\",\"displayName\":\"collect-pod-logs\",\"type\":\"DAG\",\"templateName\":\"system-container-executor\",\"templateScope\":\"local/pod-log-to-github-qv4f5\",\"phase\":\"Failed\",\"boundaryID\":\"pod-log-to-github-qv4f5-2214199526\",\"startedAt\":\"2025-05-07T05:44:13Z\",\"finishedAt\":\"2025-05-07T05:44:47Z\",\"progress\":\"0/1\",\"resourcesDuration\":{\"cpu\":26,\"memory\":26},\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"bitnami/kubectl:1.26\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"e217d06b-0721-4a9b-99c2-bc2124e299ff\\\",\\\"--execution_id\\\",\\\"34\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{\\\\\\\"parameterValues\\\\\\\":{\\\\\\\"namespace\\\\\\\":\\\\\\\"istio-system\\\\\\\", \\\\\\\"pod_name\\\\\\\":\\\\\\\"toolbox\\\\\\\"}}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/e217d06b-0721-4a9b-99c2-bc2124e299ff/collect-pod-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"inputDefinitions\\\\\\\":{\\\\\\\"parameters\\\\\\\":{\\\\\\\"namespace\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}, \\\\\\\"pod_name\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}}}, \\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-pod-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_pod_logs(\\\\n    namespace: str,\\\\n    pod_name: str,\\\\n    output_report: Output[Dataset]\\\\n):\\\\n    import subprocess\\\\n\\\\n    describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n    logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n    events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n\\\\n    with open(output_report.path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(\\\\\\\"# 장애 복기 리포트\\\\\\\\n\\\\\\\")\\\\n        f.write(f\\\\\\\"**Pod**: {pod_name}\\\\\\\\n\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## describe\\\\\\\\n```text\\\\\\\\n\\\\\\\" + describe + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## logs\\\\\\\\n```text\\\\\\\\n\\\\\\\" + logs + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## events\\\\\\\\n```text\\\\\\\\n\\\\\\\" + events + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_pod_logs\\\"],\\\"resources\\\":{}}]}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"false\"}]},\"children\":[\"pod-log-to-github-qv4f5-742416059\"],\"outboundNodes\":[\"pod-log-to-github-qv4f5-742416059\"]},\"pod-log-to-github-qv4f5-2214199526\":{\"id\":\"pod-log-to-github-qv4f5-2214199526\",\"name\":\"pod-log-to-github-qv4f5.root\",\"displayName\":\"root\",\"type\":\"DAG\",\"templateName\":\"root\",\"templateScope\":\"local/pod-log-to-github-qv4f5\",\"phase\":\"Failed\",\"boundaryID\":\"pod-log-to-github-qv4f5\",\"startedAt\":\"2025-05-07T05:44:03Z\",\"finishedAt\":\"2025-05-07T05:44:47Z\",\"progress\":\"1/2\",\"resourcesDuration\":{\"cpu\":31,\"memory\":30},\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"33\"}]},\"children\":[\"pod-log-to-github-qv4f5-4185791085\"],\"outboundNodes\":[\"pod-log-to-github-qv4f5-1172358263\"]},\"pod-log-to-github-qv4f5-3776246499\":{\"id\":\"pod-log-to-github-qv4f5-3776246499\",\"name\":\"pod-log-to-github-qv4f5.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-qv4f5\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-qv4f5\",\"startedAt\":\"2025-05-07T05:43:53Z\",\"finishedAt\":\"2025-05-07T05:43:56Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":4,\"memory\":3},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-pod-logs\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-pod-logs\\\"},\\\"inputs\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"componentInputParameter\\\":\\\"namespace\\\"},\\\"pod_name\\\":{\\\"componentInputParameter\\\":\\\"pod_name\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-pod-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-pod-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-pod-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"namespace\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"pod_name\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\",\\\"github_token\\\":\\\"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\\\",\\\"namespace\\\":\\\"istio-system\\\",\\\"pod_name\\\":\\\"toolbox\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"value\":\"33\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"value\":\"0\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-qv4f5/2025/05/07/pod-log-to-github-qv4f5-system-dag-driver-3776246499/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-qv4f5-2214199526\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-qv4f5-4185791085\":{\"id\":\"pod-log-to-github-qv4f5-4185791085\",\"name\":\"pod-log-to-github-qv4f5.root.collect-pod-logs-driver\",\"displayName\":\"collect-pod-logs-driver\",\"type\":\"Pod\",\"templateName\":\"system-container-driver\",\"templateScope\":\"local/pod-log-to-github-qv4f5\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-qv4f5-2214199526\",\"startedAt\":\"2025-05-07T05:44:03Z\",\"finishedAt\":\"2025-05-07T05:44:06Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":5,\"memory\":4},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"executorLabel\\\":\\\"exec-collect-pod-logs\\\",\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"pod_name\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}},\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-pod-logs\\\"},\\\"inputs\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"componentInputParameter\\\":\\\"namespace\\\"},\\\"pod_name\\\":{\\\"componentInputParameter\\\":\\\"pod_name\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-pod-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_pod_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_pod_logs(\\\\n    namespace: str,\\\\n    pod_name: str,\\\\n    output_report: Output[Dataset]\\\\n):\\\\n    import subprocess\\\\n\\\\n    describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n    logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n    events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n\\\\n    with open(output_report.path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(\\\\\\\"# 장애 복기 리포트\\\\\\\\n\\\\\\\")\\\\n        f.write(f\\\\\\\"**Pod**: {pod_name}\\\\\\\\n\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## describe\\\\\\\\n```text\\\\\\\\n\\\\\\\" + describe + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## logs\\\\\\\\n```text\\\\\\\\n\\\\\\\" + logs + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## events\\\\\\\\n```text\\\\\\\\n\\\\\\\" + events + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"bitnami/kubectl:1.26\\\"}\"},{\"name\":\"parent-dag-id\",\"value\":\"33\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\",\"value\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"bitnami/kubectl:1.26\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"e217d06b-0721-4a9b-99c2-bc2124e299ff\\\",\\\"--execution_id\\\",\\\"34\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{\\\\\\\"parameterValues\\\\\\\":{\\\\\\\"namespace\\\\\\\":\\\\\\\"istio-system\\\\\\\", \\\\\\\"pod_name\\\\\\\":\\\\\\\"toolbox\\\\\\\"}}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/e217d06b-0721-4a9b-99c2-bc2124e299ff/collect-pod-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"inputDefinitions\\\\\\\":{\\\\\\\"parameters\\\\\\\":{\\\\\\\"namespace\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}, \\\\\\\"pod_name\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}}}, \\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-pod-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_pod_logs(\\\\n    namespace: str,\\\\n    pod_name: str,\\\\n    output_report: Output[Dataset]\\\\n):\\\\n    import subprocess\\\\n\\\\n    describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n    logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n    events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n\\\\n    with open(output_report.path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(\\\\\\\"# 장애 복기 리포트\\\\\\\\n\\\\\\\")\\\\n        f.write(f\\\\\\\"**Pod**: {pod_name}\\\\\\\\n\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## describe\\\\\\\\n```text\\\\\\\\n\\\\\\\" + describe + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## logs\\\\\\\\n```text\\\\\\\\n\\\\\\\" + logs + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## events\\\\\\\\n```text\\\\\\\\n\\\\\\\" + events + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_pod_logs\\\"],\\\"resources\\\":{}}]}\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-qv4f5/2025/05/07/pod-log-to-github-qv4f5-system-container-driver-4185791085/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-qv4f5-2146131056\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-qv4f5-742416059\":{\"id\":\"pod-log-to-github-qv4f5-742416059\",\"name\":\"pod-log-to-github-qv4f5.root.collect-pod-logs.executor\",\"displayName\":\"executor\",\"type\":\"Pod\",\"templateName\":\"system-container-impl\",\"templateScope\":\"local/pod-log-to-github-qv4f5\",\"phase\":\"Failed\",\"boundaryID\":\"pod-log-to-github-qv4f5-2146131056\",\"message\":\"Error (exit code 1)\",\"startedAt\":\"2025-05-07T05:44:13Z\",\"finishedAt\":\"2025-05-07T05:44:39Z\",\"progress\":\"0/1\",\"resourcesDuration\":{\"cpu\":26,\"memory\":26},\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"bitnami/kubectl:1.26\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"e217d06b-0721-4a9b-99c2-bc2124e299ff\\\",\\\"--execution_id\\\",\\\"34\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{\\\\\\\"parameterValues\\\\\\\":{\\\\\\\"namespace\\\\\\\":\\\\\\\"istio-system\\\\\\\", \\\\\\\"pod_name\\\\\\\":\\\\\\\"toolbox\\\\\\\"}}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/e217d06b-0721-4a9b-99c2-bc2124e299ff/collect-pod-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"inputDefinitions\\\\\\\":{\\\\\\\"parameters\\\\\\\":{\\\\\\\"namespace\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}, \\\\\\\"pod_name\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}}}, \\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-pod-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_pod_logs(\\\\n    namespace: str,\\\\n    pod_name: str,\\\\n    output_report: Output[Dataset]\\\\n):\\\\n    import subprocess\\\\n\\\\n    describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n    logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n    events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n\\\\n    with open(output_report.path, \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(\\\\\\\"# 장애 복기 리포트\\\\\\\\n\\\\\\\")\\\\n        f.write(f\\\\\\\"**Pod**: {pod_name}\\\\\\\\n\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## describe\\\\\\\\n```text\\\\\\\\n\\\\\\\" + describe + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## logs\\\\\\\\n```text\\\\\\\\n\\\\\\\" + logs + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## events\\\\\\\\n```text\\\\\\\\n\\\\\\\" + events + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_pod_logs\\\"],\\\"resources\\\":{}}]}\"}]},\"outputs\":{\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-qv4f5/2025/05/07/pod-log-to-github-qv4f5-system-container-impl-742416059/main.log\"}}],\"exitCode\":\"1\"},\"children\":[\"pod-log-to-github-qv4f5-1246834064\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"}},\"conditions\":[{\"type\":\"PodRunning\",\"status\":\"False\"},{\"type\":\"Completed\",\"status\":\"True\"}],\"resourcesDuration\":{\"cpu\":35,\"memory\":33},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = NotFound desc = Failed to report workflow: NotFoundError: Failed to add PersistedFinalState label to workflow pod-log-to-github-qv4f5: workflows.argoproj.io \"pod-log-to-github-qv4f5\" not found"
time="2025-05-08T05:44:53Z" level=error msg="Permanent failure while syncing resource (kubeflow-user-example-com/pod-log-to-github-qv4f5): CustomError (code: 1): Workflow (kubeflow-user-example-com/pod-log-to-github-qv4f5) in work queue no longer exists: CustomError (code: 2): Error retrieving workflow (pod-log-to-github-qv4f5) in namespace (kubeflow-user-example-com): Error retrieving workflow (pod-log-to-github-qv4f5) in namespace (kubeflow-user-example-com): workflow.argoproj.io \"pod-log-to-github-qv4f5\" not found: workflow.argoproj.io \"pod-log-to-github-qv4f5\" not found: Error retrieving workflow (pod-log-to-github-qv4f5) in namespace (kubeflow-user-example-com): workflow.argoproj.io \"pod-log-to-github-qv4f5\" not found: workflow.argoproj.io \"pod-log-to-github-qv4f5\" not found: CustomError (code: 2): Error retrieving workflow (pod-log-to-github-qv4f5) in namespace (kubeflow-user-example-com): Error retrieving workflow (pod-log-to-github-qv4f5) in namespace (kubeflow-user-example-com): workflow.argoproj.io \"pod-log-to-github-qv4f5\" not found: workflow.argoproj.io \"pod-log-to-github-qv4f5\" not found: Error retrieving workflow (pod-log-to-github-qv4f5) in namespace (kubeflow-user-example-com): workflow.argoproj.io \"pod-log-to-github-qv4f5\" not found: workflow.argoproj.io \"pod-log-to-github-qv4f5\" not found"
time="2025-05-08T05:56:53Z" level=error msg="Permanent failure while syncing resource (kubeflow-user-example-com/pod-log-to-github-gfhgz): CustomError (code: 1): Syncing Workflow (pod-log-to-github-gfhgz): permanent failure: CustomError (code: 1): Error while reporting workflow resource (code: NotFound, message: Failed to report workflow: NotFoundError: Failed to add PersistedFinalState label to workflow pod-log-to-github-gfhgz: workflows.argoproj.io \"pod-log-to-github-gfhgz\" not found): rpc error: code = NotFound desc = Failed to report workflow: NotFoundError: Failed to add PersistedFinalState label to workflow pod-log-to-github-gfhgz: workflows.argoproj.io \"pod-log-to-github-gfhgz\" not found, {\"metadata\":{\"name\":\"pod-log-to-github-gfhgz\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"6c9c801d-970d-4f2d-824a-7c15802d2912\",\"resourceVersion\":\"6051395\",\"generation\":9,\"creationTimestamp\":\"2025-05-07T05:49:00Z\",\"labels\":{\"pipeline/persistedFinalState\":\"true\",\"pipeline/runid\":\"7831eb42-18ea-4116-9d33-c327f3efec4d\",\"workflows.argoproj.io/completed\":\"true\",\"workflows.argoproj.io/phase\":\"Failed\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-pod-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-pod-logs\\\",\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"pod_name\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}},\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-pod-logs\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-pod-logs\\\"},\\\"inputs\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"componentInputParameter\\\":\\\"namespace\\\"},\\\"pod_name\\\":{\\\"componentInputParameter\\\":\\\"pod_name\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-pod-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-pod-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-pod-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"namespace\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"pod_name\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-pod-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_pod_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_pod_logs(\\\\n    namespace: str,\\\\n    pod_name: str,\\\\n    output_report: Output[Dataset]\\\\n):\\\\n    import subprocess\\\\n\\\\n    describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n    logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n    events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n\\\\n    with open(\\\\\\\"/tmp/output_report.md\\\\\\\", \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(\\\\\\\"# 장애 복기 리포트\\\\\\\\n\\\\\\\")\\\\n        f.write(f\\\\\\\"**Pod**: {pod_name}\\\\\\\\n\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## describe\\\\\\\\n```text\\\\\\\\n\\\\\\\" + describe + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## logs\\\\\\\\n```text\\\\\\\\n\\\\\\\" + logs + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## events\\\\\\\\n```text\\\\\\\\n\\\\\\\" + events + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n\\\\n    output_report.path = \\\\\\\"/tmp/output_report.md\\\\\\\"\\\\n\\\\n\\\"],\\\"image\\\":\\\"bitnami/kubectl:1.26\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str = \\\\\\\"main\\\\\\\"\\\\n):\\\\n    import os\\\\n    import subprocess\\\\n\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_file.path)\\\\n    if not os.path.exists(report_file.path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_file.path} does not exist\\\\\\\")\\\\n\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", report_path, os.path.join(repo_dir, \\\\\\\"report.md\\\\\\\")], check=True)\\\\n\\\\n    # Git user config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", \\\\\\\"Add report\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(\\\\\\\"Report committed and pushed to GitHub.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Run of pod_log_pipeline_version_at_2025-05-07T05:48:39.055Z (2d762)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-07T05:56:41Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/completed\":{},\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}},{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-07T05:56:43Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-pod-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-pod-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/persistedFinalState\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"7831eb42-18ea-4116-9d33-c327f3efec4d\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-pod-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-pod-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-pod-logs\\\"},\\\"inputs\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"componentInputParameter\\\":\\\"namespace\\\"},\\\"pod_name\\\":{\\\"componentInputParameter\\\":\\\"pod_name\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-pod-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-pod-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-pod-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-pod-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-pod-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-pod-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-pod-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-pod-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-pod-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"7831eb42-18ea-4116-9d33-c327f3efec4d\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\",\\\"github_token\\\":\\\"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\\\",\\\"namespace\\\":\\\"istio-system\\\",\\\"pod_name\\\":\\\"toolbox\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"activeDeadlineSeconds\":0,\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"7831eb42-18ea-4116-9d33-c327f3efec4d\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Failed\",\"startedAt\":\"2025-05-07T05:49:01Z\",\"finishedAt\":\"2025-05-07T05:56:41Z\",\"progress\":\"2/3\",\"nodes\":{\"pod-log-to-github-gfhgz\":{\"id\":\"pod-log-to-github-gfhgz\",\"name\":\"pod-log-to-github-gfhgz\",\"displayName\":\"pod-log-to-github-gfhgz\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-gfhgz\",\"phase\":\"Failed\",\"startedAt\":\"2025-05-07T05:49:01Z\",\"finishedAt\":\"2025-05-07T05:56:41Z\",\"progress\":\"0/1\",\"children\":[\"pod-log-to-github-gfhgz-1021093953\"],\"outboundNodes\":[\"pod-log-to-github-gfhgz-733409036\"]},\"pod-log-to-github-gfhgz-1021093953\":{\"id\":\"pod-log-to-github-gfhgz-1021093953\",\"name\":\"pod-log-to-github-gfhgz.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-gfhgz\",\"phase\":\"Failed\",\"boundaryID\":\"pod-log-to-github-gfhgz\",\"message\":\"Step exceeded its deadline\",\"startedAt\":\"2025-05-07T05:52:42Z\",\"finishedAt\":\"2025-05-07T05:56:41Z\",\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-pod-logs\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-pod-logs\\\"},\\\"inputs\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"componentInputParameter\\\":\\\"namespace\\\"},\\\"pod_name\\\":{\\\"componentInputParameter\\\":\\\"pod_name\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-pod-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-pod-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-pod-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"namespace\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"pod_name\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\",\\\"github_token\\\":\\\"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\\\",\\\"namespace\\\":\\\"istio-system\\\",\\\"pod_name\\\":\\\"toolbox\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]},\"children\":[\"pod-log-to-github-gfhgz-733409036\"]},\"pod-log-to-github-gfhgz-733409036\":{\"id\":\"pod-log-to-github-gfhgz-733409036\",\"name\":\"pod-log-to-github-gfhgz.root\",\"displayName\":\"root\",\"type\":\"Skipped\",\"templateName\":\"root\",\"templateScope\":\"local/pod-log-to-github-gfhgz\",\"phase\":\"Omitted\",\"boundaryID\":\"pod-log-to-github-gfhgz\",\"message\":\"omitted: depends condition not met\",\"startedAt\":\"2025-05-07T05:56:41Z\",\"finishedAt\":\"2025-05-07T05:56:41Z\"},\"pod-log-to-github-gfhgz-root-733409036\":{\"id\":\"pod-log-to-github-gfhgz-733409036\",\"name\":\"pod-log-to-github-gfhgz.root\",\"displayName\":\"root\",\"type\":\"DAG\",\"templateName\":\"root\",\"templateScope\":\"local/pod-log-to-github-gfhgz\",\"phase\":\"Running\",\"boundaryID\":\"pod-log-to-github-gfhgz\",\"startedAt\":\"2025-05-07T05:49:11Z\",\"finishedAt\":null,\"progress\":\"1/2\",\"resourcesDuration\":{\"cpu\":7,\"memory\":6},\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"35\"}]},\"children\":[\"pod-log-to-github-gfhgz-71102127\"],\"outboundNodes\":[\"pod-log-to-github-gfhgz-2042521421\"]},\"pod-log-to-github-gfhgz-system-container-driver-71102127\":{\"id\":\"pod-log-to-github-gfhgz-71102127\",\"name\":\"pod-log-to-github-gfhgz.root.collect-pod-logs-driver\",\"displayName\":\"collect-pod-logs-driver\",\"type\":\"Pod\",\"templateName\":\"system-container-driver\",\"templateScope\":\"local/pod-log-to-github-gfhgz\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-gfhgz-733409036\",\"startedAt\":\"2025-05-07T05:49:11Z\",\"finishedAt\":\"2025-05-07T05:49:14Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":3,\"memory\":2},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"executorLabel\\\":\\\"exec-collect-pod-logs\\\",\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"pod_name\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}},\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-pod-logs\\\"},\\\"inputs\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"componentInputParameter\\\":\\\"namespace\\\"},\\\"pod_name\\\":{\\\"componentInputParameter\\\":\\\"pod_name\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-pod-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_pod_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_pod_logs(\\\\n    namespace: str,\\\\n    pod_name: str,\\\\n    output_report: Output[Dataset]\\\\n):\\\\n    import subprocess\\\\n\\\\n    describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n    logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n    events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n\\\\n    with open(\\\\\\\"/tmp/output_report.md\\\\\\\", \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(\\\\\\\"# 장애 복기 리포트\\\\\\\\n\\\\\\\")\\\\n        f.write(f\\\\\\\"**Pod**: {pod_name}\\\\\\\\n\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## describe\\\\\\\\n```text\\\\\\\\n\\\\\\\" + describe + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## logs\\\\\\\\n```text\\\\\\\\n\\\\\\\" + logs + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## events\\\\\\\\n```text\\\\\\\\n\\\\\\\" + events + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n\\\\n    output_report.path = \\\\\\\"/tmp/output_report.md\\\\\\\"\\\\n\\\\n\\\"],\\\"image\\\":\\\"bitnami/kubectl:1.26\\\"}\"},{\"name\":\"parent-dag-id\",\"value\":\"35\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\",\"value\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"bitnami/kubectl:1.26\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"7831eb42-18ea-4116-9d33-c327f3efec4d\\\",\\\"--execution_id\\\",\\\"36\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{\\\\\\\"parameterValues\\\\\\\":{\\\\\\\"namespace\\\\\\\":\\\\\\\"istio-system\\\\\\\", \\\\\\\"pod_name\\\\\\\":\\\\\\\"toolbox\\\\\\\"}}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/7831eb42-18ea-4116-9d33-c327f3efec4d/collect-pod-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"inputDefinitions\\\\\\\":{\\\\\\\"parameters\\\\\\\":{\\\\\\\"namespace\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}, \\\\\\\"pod_name\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}}}, \\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-pod-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_pod_logs(\\\\n    namespace: str,\\\\n    pod_name: str,\\\\n    output_report: Output[Dataset]\\\\n):\\\\n    import subprocess\\\\n\\\\n    describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n    logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n    events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n\\\\n    with open(\\\\\\\"/tmp/output_report.md\\\\\\\", \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(\\\\\\\"# 장애 복기 리포트\\\\\\\\n\\\\\\\")\\\\n        f.write(f\\\\\\\"**Pod**: {pod_name}\\\\\\\\n\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## describe\\\\\\\\n```text\\\\\\\\n\\\\\\\" + describe + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## logs\\\\\\\\n```text\\\\\\\\n\\\\\\\" + logs + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## events\\\\\\\\n```text\\\\\\\\n\\\\\\\" + events + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n\\\\n    output_report.path = \\\\\\\"/tmp/output_report.md\\\\\\\"\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_pod_logs\\\"],\\\"resources\\\":{}}]}\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-gfhgz/2025/05/07/pod-log-to-github-gfhgz-system-container-driver-71102127/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-gfhgz-73901370\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-gfhgz-system-container-executor-73901370\":{\"id\":\"pod-log-to-github-gfhgz-73901370\",\"name\":\"pod-log-to-github-gfhgz.root.collect-pod-logs\",\"displayName\":\"collect-pod-logs\",\"type\":\"DAG\",\"templateName\":\"system-container-executor\",\"templateScope\":\"local/pod-log-to-github-gfhgz\",\"phase\":\"Running\",\"boundaryID\":\"pod-log-to-github-gfhgz-733409036\",\"startedAt\":\"2025-05-07T05:49:21Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"resourcesDuration\":{\"cpu\":4,\"memory\":4},\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"bitnami/kubectl:1.26\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"7831eb42-18ea-4116-9d33-c327f3efec4d\\\",\\\"--execution_id\\\",\\\"36\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{\\\\\\\"parameterValues\\\\\\\":{\\\\\\\"namespace\\\\\\\":\\\\\\\"istio-system\\\\\\\", \\\\\\\"pod_name\\\\\\\":\\\\\\\"toolbox\\\\\\\"}}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/7831eb42-18ea-4116-9d33-c327f3efec4d/collect-pod-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"inputDefinitions\\\\\\\":{\\\\\\\"parameters\\\\\\\":{\\\\\\\"namespace\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}, \\\\\\\"pod_name\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}}}, \\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-pod-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_pod_logs(\\\\n    namespace: str,\\\\n    pod_name: str,\\\\n    output_report: Output[Dataset]\\\\n):\\\\n    import subprocess\\\\n\\\\n    describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n    logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n    events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n\\\\n    with open(\\\\\\\"/tmp/output_report.md\\\\\\\", \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(\\\\\\\"# 장애 복기 리포트\\\\\\\\n\\\\\\\")\\\\n        f.write(f\\\\\\\"**Pod**: {pod_name}\\\\\\\\n\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## describe\\\\\\\\n```text\\\\\\\\n\\\\\\\" + describe + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## logs\\\\\\\\n```text\\\\\\\\n\\\\\\\" + logs + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## events\\\\\\\\n```text\\\\\\\\n\\\\\\\" + events + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n\\\\n    output_report.path = \\\\\\\"/tmp/output_report.md\\\\\\\"\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_pod_logs\\\"],\\\"resources\\\":{}}]}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"false\"}]},\"children\":[\"pod-log-to-github-gfhgz-3552771745\"],\"outboundNodes\":[\"pod-log-to-github-gfhgz-3552771745\"]},\"pod-log-to-github-gfhgz-system-dag-driver-1021093953\":{\"id\":\"pod-log-to-github-gfhgz-1021093953\",\"name\":\"pod-log-to-github-gfhgz.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-gfhgz\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-gfhgz\",\"startedAt\":\"2025-05-07T05:49:01Z\",\"finishedAt\":\"2025-05-07T05:49:04Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":3,\"memory\":2},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-pod-logs\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-pod-logs\\\"},\\\"inputs\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"componentInputParameter\\\":\\\"namespace\\\"},\\\"pod_name\\\":{\\\"componentInputParameter\\\":\\\"pod_name\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-pod-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-pod-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-pod-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"namespace\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"pod_name\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\",\\\"github_token\\\":\\\"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\\\",\\\"namespace\\\":\\\"istio-system\\\",\\\"pod_name\\\":\\\"toolbox\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"value\":\"35\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"value\":\"0\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-gfhgz/2025/05/07/pod-log-to-github-gfhgz-system-dag-driver-1021093953/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-gfhgz-733409036\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"}},\"conditions\":[{\"type\":\"PodRunning\",\"status\":\"False\"},{\"type\":\"Completed\",\"status\":\"True\"}],\"resourcesDuration\":{\"cpu\":6,\"memory\":4},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = NotFound desc = Failed to report workflow: NotFoundError: Failed to add PersistedFinalState label to workflow pod-log-to-github-gfhgz: workflows.argoproj.io \"pod-log-to-github-gfhgz\" not found: CustomError (code: 1): Error while reporting workflow resource (code: NotFound, message: Failed to report workflow: NotFoundError: Failed to add PersistedFinalState label to workflow pod-log-to-github-gfhgz: workflows.argoproj.io \"pod-log-to-github-gfhgz\" not found): rpc error: code = NotFound desc = Failed to report workflow: NotFoundError: Failed to add PersistedFinalState label to workflow pod-log-to-github-gfhgz: workflows.argoproj.io \"pod-log-to-github-gfhgz\" not found, {\"metadata\":{\"name\":\"pod-log-to-github-gfhgz\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"6c9c801d-970d-4f2d-824a-7c15802d2912\",\"resourceVersion\":\"6051395\",\"generation\":9,\"creationTimestamp\":\"2025-05-07T05:49:00Z\",\"labels\":{\"pipeline/persistedFinalState\":\"true\",\"pipeline/runid\":\"7831eb42-18ea-4116-9d33-c327f3efec4d\",\"workflows.argoproj.io/completed\":\"true\",\"workflows.argoproj.io/phase\":\"Failed\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-pod-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-pod-logs\\\",\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"pod_name\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}},\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-pod-logs\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-pod-logs\\\"},\\\"inputs\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"componentInputParameter\\\":\\\"namespace\\\"},\\\"pod_name\\\":{\\\"componentInputParameter\\\":\\\"pod_name\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-pod-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-pod-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-pod-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"namespace\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"pod_name\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-pod-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_pod_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_pod_logs(\\\\n    namespace: str,\\\\n    pod_name: str,\\\\n    output_report: Output[Dataset]\\\\n):\\\\n    import subprocess\\\\n\\\\n    describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n    logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n    events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n\\\\n    with open(\\\\\\\"/tmp/output_report.md\\\\\\\", \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(\\\\\\\"# 장애 복기 리포트\\\\\\\\n\\\\\\\")\\\\n        f.write(f\\\\\\\"**Pod**: {pod_name}\\\\\\\\n\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## describe\\\\\\\\n```text\\\\\\\\n\\\\\\\" + describe + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## logs\\\\\\\\n```text\\\\\\\\n\\\\\\\" + logs + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## events\\\\\\\\n```text\\\\\\\\n\\\\\\\" + events + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n\\\\n    output_report.path = \\\\\\\"/tmp/output_report.md\\\\\\\"\\\\n\\\\n\\\"],\\\"image\\\":\\\"bitnami/kubectl:1.26\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str = \\\\\\\"main\\\\\\\"\\\\n):\\\\n    import os\\\\n    import subprocess\\\\n\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_file.path)\\\\n    if not os.path.exists(report_file.path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_file.path} does not exist\\\\\\\")\\\\n\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", report_path, os.path.join(repo_dir, \\\\\\\"report.md\\\\\\\")], check=True)\\\\n\\\\n    # Git user config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", \\\\\\\"Add report\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(\\\\\\\"Report committed and pushed to GitHub.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Run of pod_log_pipeline_version_at_2025-05-07T05:48:39.055Z (2d762)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-07T05:56:41Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/completed\":{},\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}},{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-07T05:56:43Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-pod-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-pod-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/persistedFinalState\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"7831eb42-18ea-4116-9d33-c327f3efec4d\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-pod-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-pod-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-pod-logs\\\"},\\\"inputs\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"componentInputParameter\\\":\\\"namespace\\\"},\\\"pod_name\\\":{\\\"componentInputParameter\\\":\\\"pod_name\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-pod-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-pod-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-pod-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-pod-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-pod-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-pod-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-pod-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-pod-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-pod-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"7831eb42-18ea-4116-9d33-c327f3efec4d\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\",\\\"github_token\\\":\\\"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\\\",\\\"namespace\\\":\\\"istio-system\\\",\\\"pod_name\\\":\\\"toolbox\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"activeDeadlineSeconds\":0,\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"7831eb42-18ea-4116-9d33-c327f3efec4d\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Failed\",\"startedAt\":\"2025-05-07T05:49:01Z\",\"finishedAt\":\"2025-05-07T05:56:41Z\",\"progress\":\"2/3\",\"nodes\":{\"pod-log-to-github-gfhgz\":{\"id\":\"pod-log-to-github-gfhgz\",\"name\":\"pod-log-to-github-gfhgz\",\"displayName\":\"pod-log-to-github-gfhgz\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-gfhgz\",\"phase\":\"Failed\",\"startedAt\":\"2025-05-07T05:49:01Z\",\"finishedAt\":\"2025-05-07T05:56:41Z\",\"progress\":\"0/1\",\"children\":[\"pod-log-to-github-gfhgz-1021093953\"],\"outboundNodes\":[\"pod-log-to-github-gfhgz-733409036\"]},\"pod-log-to-github-gfhgz-1021093953\":{\"id\":\"pod-log-to-github-gfhgz-1021093953\",\"name\":\"pod-log-to-github-gfhgz.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-gfhgz\",\"phase\":\"Failed\",\"boundaryID\":\"pod-log-to-github-gfhgz\",\"message\":\"Step exceeded its deadline\",\"startedAt\":\"2025-05-07T05:52:42Z\",\"finishedAt\":\"2025-05-07T05:56:41Z\",\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-pod-logs\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-pod-logs\\\"},\\\"inputs\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"componentInputParameter\\\":\\\"namespace\\\"},\\\"pod_name\\\":{\\\"componentInputParameter\\\":\\\"pod_name\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-pod-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-pod-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-pod-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"namespace\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"pod_name\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\",\\\"github_token\\\":\\\"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\\\",\\\"namespace\\\":\\\"istio-system\\\",\\\"pod_name\\\":\\\"toolbox\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]},\"children\":[\"pod-log-to-github-gfhgz-733409036\"]},\"pod-log-to-github-gfhgz-733409036\":{\"id\":\"pod-log-to-github-gfhgz-733409036\",\"name\":\"pod-log-to-github-gfhgz.root\",\"displayName\":\"root\",\"type\":\"Skipped\",\"templateName\":\"root\",\"templateScope\":\"local/pod-log-to-github-gfhgz\",\"phase\":\"Omitted\",\"boundaryID\":\"pod-log-to-github-gfhgz\",\"message\":\"omitted: depends condition not met\",\"startedAt\":\"2025-05-07T05:56:41Z\",\"finishedAt\":\"2025-05-07T05:56:41Z\"},\"pod-log-to-github-gfhgz-root-733409036\":{\"id\":\"pod-log-to-github-gfhgz-733409036\",\"name\":\"pod-log-to-github-gfhgz.root\",\"displayName\":\"root\",\"type\":\"DAG\",\"templateName\":\"root\",\"templateScope\":\"local/pod-log-to-github-gfhgz\",\"phase\":\"Running\",\"boundaryID\":\"pod-log-to-github-gfhgz\",\"startedAt\":\"2025-05-07T05:49:11Z\",\"finishedAt\":null,\"progress\":\"1/2\",\"resourcesDuration\":{\"cpu\":7,\"memory\":6},\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"35\"}]},\"children\":[\"pod-log-to-github-gfhgz-71102127\"],\"outboundNodes\":[\"pod-log-to-github-gfhgz-2042521421\"]},\"pod-log-to-github-gfhgz-system-container-driver-71102127\":{\"id\":\"pod-log-to-github-gfhgz-71102127\",\"name\":\"pod-log-to-github-gfhgz.root.collect-pod-logs-driver\",\"displayName\":\"collect-pod-logs-driver\",\"type\":\"Pod\",\"templateName\":\"system-container-driver\",\"templateScope\":\"local/pod-log-to-github-gfhgz\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-gfhgz-733409036\",\"startedAt\":\"2025-05-07T05:49:11Z\",\"finishedAt\":\"2025-05-07T05:49:14Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":3,\"memory\":2},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"executorLabel\\\":\\\"exec-collect-pod-logs\\\",\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"pod_name\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}},\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-pod-logs\\\"},\\\"inputs\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"componentInputParameter\\\":\\\"namespace\\\"},\\\"pod_name\\\":{\\\"componentInputParameter\\\":\\\"pod_name\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-pod-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_pod_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_pod_logs(\\\\n    namespace: str,\\\\n    pod_name: str,\\\\n    output_report: Output[Dataset]\\\\n):\\\\n    import subprocess\\\\n\\\\n    describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n    logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n    events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n\\\\n    with open(\\\\\\\"/tmp/output_report.md\\\\\\\", \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(\\\\\\\"# 장애 복기 리포트\\\\\\\\n\\\\\\\")\\\\n        f.write(f\\\\\\\"**Pod**: {pod_name}\\\\\\\\n\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## describe\\\\\\\\n```text\\\\\\\\n\\\\\\\" + describe + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## logs\\\\\\\\n```text\\\\\\\\n\\\\\\\" + logs + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## events\\\\\\\\n```text\\\\\\\\n\\\\\\\" + events + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n\\\\n    output_report.path = \\\\\\\"/tmp/output_report.md\\\\\\\"\\\\n\\\\n\\\"],\\\"image\\\":\\\"bitnami/kubectl:1.26\\\"}\"},{\"name\":\"parent-dag-id\",\"value\":\"35\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\",\"value\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"bitnami/kubectl:1.26\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"7831eb42-18ea-4116-9d33-c327f3efec4d\\\",\\\"--execution_id\\\",\\\"36\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{\\\\\\\"parameterValues\\\\\\\":{\\\\\\\"namespace\\\\\\\":\\\\\\\"istio-system\\\\\\\", \\\\\\\"pod_name\\\\\\\":\\\\\\\"toolbox\\\\\\\"}}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/7831eb42-18ea-4116-9d33-c327f3efec4d/collect-pod-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"inputDefinitions\\\\\\\":{\\\\\\\"parameters\\\\\\\":{\\\\\\\"namespace\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}, \\\\\\\"pod_name\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}}}, \\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-pod-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_pod_logs(\\\\n    namespace: str,\\\\n    pod_name: str,\\\\n    output_report: Output[Dataset]\\\\n):\\\\n    import subprocess\\\\n\\\\n    describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n    logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n    events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n\\\\n    with open(\\\\\\\"/tmp/output_report.md\\\\\\\", \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(\\\\\\\"# 장애 복기 리포트\\\\\\\\n\\\\\\\")\\\\n        f.write(f\\\\\\\"**Pod**: {pod_name}\\\\\\\\n\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## describe\\\\\\\\n```text\\\\\\\\n\\\\\\\" + describe + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## logs\\\\\\\\n```text\\\\\\\\n\\\\\\\" + logs + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## events\\\\\\\\n```text\\\\\\\\n\\\\\\\" + events + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n\\\\n    output_report.path = \\\\\\\"/tmp/output_report.md\\\\\\\"\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_pod_logs\\\"],\\\"resources\\\":{}}]}\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-gfhgz/2025/05/07/pod-log-to-github-gfhgz-system-container-driver-71102127/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-gfhgz-73901370\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-gfhgz-system-container-executor-73901370\":{\"id\":\"pod-log-to-github-gfhgz-73901370\",\"name\":\"pod-log-to-github-gfhgz.root.collect-pod-logs\",\"displayName\":\"collect-pod-logs\",\"type\":\"DAG\",\"templateName\":\"system-container-executor\",\"templateScope\":\"local/pod-log-to-github-gfhgz\",\"phase\":\"Running\",\"boundaryID\":\"pod-log-to-github-gfhgz-733409036\",\"startedAt\":\"2025-05-07T05:49:21Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"resourcesDuration\":{\"cpu\":4,\"memory\":4},\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"bitnami/kubectl:1.26\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"7831eb42-18ea-4116-9d33-c327f3efec4d\\\",\\\"--execution_id\\\",\\\"36\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{\\\\\\\"parameterValues\\\\\\\":{\\\\\\\"namespace\\\\\\\":\\\\\\\"istio-system\\\\\\\", \\\\\\\"pod_name\\\\\\\":\\\\\\\"toolbox\\\\\\\"}}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/7831eb42-18ea-4116-9d33-c327f3efec4d/collect-pod-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"inputDefinitions\\\\\\\":{\\\\\\\"parameters\\\\\\\":{\\\\\\\"namespace\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}, \\\\\\\"pod_name\\\\\\\":{\\\\\\\"parameterType\\\\\\\":\\\\\\\"STRING\\\\\\\"}}}, \\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-pod-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_pod_logs(\\\\n    namespace: str,\\\\n    pod_name: str,\\\\n    output_report: Output[Dataset]\\\\n):\\\\n    import subprocess\\\\n\\\\n    describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n    logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n    events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n\\\\n    with open(\\\\\\\"/tmp/output_report.md\\\\\\\", \\\\\\\"w\\\\\\\") as f:\\\\n        f.write(\\\\\\\"# 장애 복기 리포트\\\\\\\\n\\\\\\\")\\\\n        f.write(f\\\\\\\"**Pod**: {pod_name}\\\\\\\\n\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## describe\\\\\\\\n```text\\\\\\\\n\\\\\\\" + describe + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## logs\\\\\\\\n```text\\\\\\\\n\\\\\\\" + logs + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n        f.write(\\\\\\\"## events\\\\\\\\n```text\\\\\\\\n\\\\\\\" + events + \\\\\\\"\\\\\\\\n```\\\\\\\\n\\\\\\\")\\\\n\\\\n    output_report.path = \\\\\\\"/tmp/output_report.md\\\\\\\"\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_pod_logs\\\"],\\\"resources\\\":{}}]}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"false\"}]},\"children\":[\"pod-log-to-github-gfhgz-3552771745\"],\"outboundNodes\":[\"pod-log-to-github-gfhgz-3552771745\"]},\"pod-log-to-github-gfhgz-system-dag-driver-1021093953\":{\"id\":\"pod-log-to-github-gfhgz-1021093953\",\"name\":\"pod-log-to-github-gfhgz.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-gfhgz\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-gfhgz\",\"startedAt\":\"2025-05-07T05:49:01Z\",\"finishedAt\":\"2025-05-07T05:49:04Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":3,\"memory\":2},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-pod-logs\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-pod-logs\\\"},\\\"inputs\\\":{\\\"parameters\\\":{\\\"namespace\\\":{\\\"componentInputParameter\\\":\\\"namespace\\\"},\\\"pod_name\\\":{\\\"componentInputParameter\\\":\\\"pod_name\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-pod-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-pod-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-pod-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"main\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"namespace\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"pod_name\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"main\\\",\\\"github_token\\\":\\\"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\\\",\\\"namespace\\\":\\\"istio-system\\\",\\\"pod_name\\\":\\\"toolbox\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"value\":\"35\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"value\":\"0\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-gfhgz/2025/05/07/pod-log-to-github-gfhgz-system-dag-driver-1021093953/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-gfhgz-733409036\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"}},\"conditions\":[{\"type\":\"PodRunning\",\"status\":\"False\"},{\"type\":\"Completed\",\"status\":\"True\"}],\"resourcesDuration\":{\"cpu\":6,\"memory\":4},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = NotFound desc = Failed to report workflow: NotFoundError: Failed to add PersistedFinalState label to workflow pod-log-to-github-gfhgz: workflows.argoproj.io \"pod-log-to-github-gfhgz\" not found"
time="2025-05-08T05:56:53Z" level=error msg="Permanent failure while syncing resource (kubeflow-user-example-com/pod-log-to-github-gfhgz): CustomError (code: 1): Workflow (kubeflow-user-example-com/pod-log-to-github-gfhgz) in work queue no longer exists: CustomError (code: 2): Error retrieving workflow (pod-log-to-github-gfhgz) in namespace (kubeflow-user-example-com): Error retrieving workflow (pod-log-to-github-gfhgz) in namespace (kubeflow-user-example-com): workflow.argoproj.io \"pod-log-to-github-gfhgz\" not found: workflow.argoproj.io \"pod-log-to-github-gfhgz\" not found: Error retrieving workflow (pod-log-to-github-gfhgz) in namespace (kubeflow-user-example-com): workflow.argoproj.io \"pod-log-to-github-gfhgz\" not found: workflow.argoproj.io \"pod-log-to-github-gfhgz\" not found: CustomError (code: 2): Error retrieving workflow (pod-log-to-github-gfhgz) in namespace (kubeflow-user-example-com): Error retrieving workflow (pod-log-to-github-gfhgz) in namespace (kubeflow-user-example-com): workflow.argoproj.io \"pod-log-to-github-gfhgz\" not found: workflow.argoproj.io \"pod-log-to-github-gfhgz\" not found: Error retrieving workflow (pod-log-to-github-gfhgz) in namespace (kubeflow-user-example-com): workflow.argoproj.io \"pod-log-to-github-gfhgz\" not found: workflow.argoproj.io \"pod-log-to-github-gfhgz\" not found"
time="2025-05-08T06:28:53Z" level=error msg="Permanent failure while syncing resource (kubeflow-user-example-com/pod-log-to-github-jt27k): CustomError (code: 1): Syncing Workflow (pod-log-to-github-jt27k): permanent failure: CustomError (code: 1): Error while reporting workflow resource (code: NotFound, message: Failed to report workflow: Failed to report workflow name=\"pod-log-to-github-jt27k\" namespace=\"kubeflow-user-example-com\" runId=\"ecfd93e6-166e-4811-920a-fb70c4edd9f1\": Failed to fetch run ecfd93e6-166e-4811-920a-fb70c4edd9f1: ResourceNotFoundError: Run ecfd93e6-166e-4811-920a-fb70c4edd9f1 not found): rpc error: code = NotFound desc = Failed to report workflow: Failed to report workflow name=\"pod-log-to-github-jt27k\" namespace=\"kubeflow-user-example-com\" runId=\"ecfd93e6-166e-4811-920a-fb70c4edd9f1\": Failed to fetch run ecfd93e6-166e-4811-920a-fb70c4edd9f1: ResourceNotFoundError: Run ecfd93e6-166e-4811-920a-fb70c4edd9f1 not found, {\"metadata\":{\"name\":\"pod-log-to-github-jt27k\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"6320da99-118e-4c98-af55-42f7b61b864c\",\"resourceVersion\":\"6552414\",\"generation\":2,\"creationTimestamp\":\"2025-05-08T06:28:52Z\",\"labels\":{\"pipeline/runid\":\"ecfd93e6-166e-4811-920a-fb70c4edd9f1\",\"workflows.argoproj.io/phase\":\"Running\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-filtered-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"defaultValue\\\":\\\"crash\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"}}},\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\\\\\"crash\\\\\\\"):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Pod 목록 가져오기\\\\n    pods_output = subprocess.getoutput(\\\\n        \\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\"\\\\n    )\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML 시작\\\\n        f.write(f\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"utf-8\\\\\\\"\\\\u003e\\\\n            \\\\u003ctitle\\\\u003e장애 복기 리포트\\\\u003c/title\\\\u003e\\\\n            \\\\u003clink href=\\\\\\\"https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/css/bootstrap.min.css\\\\\\\" rel=\\\\\\\"stylesheet\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body {{ padding: 2rem; background-color: #f8f9fa; }}\\\\n                pre {{ background-color: #f1f1f1; padding: 1rem; border-radius: 5px; overflow-x: auto; }}\\\\n                .card {{ margin-bottom: 2rem; }}\\\\n                .keyword {{ background-color: yellow; font-weight: bold; }}\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n            \\\\u003cdiv class=\\\\\\\"container\\\\\\\"\\\\u003e\\\\n            \\\\u003ch1 class=\\\\\\\"mb-4\\\\\\\"\\\\u003e🚨 장애 복기 리포트\\\\u003c/h1\\\\u003e\\\\n            \\\\u003ch5 class=\\\\\\\"text-muted\\\\\\\"\\\\u003eKeyword: \\\\u003cspan class=\\\\\\\"badge bg-danger\\\\\\\"\\\\u003e{keyword}\\\\u003c/span\\\\u003e\\\\u003c/h5\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in describe.splitlines() if keyword in l])\\\\n            matched_logs = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in logs.splitlines() if keyword in l])\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\\\\"\\\\\\\"\\\\n                \\\\u003cdiv class=\\\\\\\"card\\\\\\\"\\\\u003e\\\\n                    \\\\u003cdiv class=\\\\\\\"card-header bg-primary text-white\\\\\\\"\\\\u003e\\\\n                        Pod: \\\\u003cstrong\\\\u003e{pod_name}\\\\u003c/strong\\\\u003e \\\\u0026nbsp;\\\\u0026nbsp;|\\\\u0026nbsp;\\\\u0026nbsp; Namespace: \\\\u003cstrong\\\\u003e{namespace}\\\\u003c/strong\\\\u003e\\\\n                    \\\\u003c/div\\\\u003e\\\\n                    \\\\u003cdiv class=\\\\\\\"card-body\\\\\\\"\\\\u003e\\\\n                \\\\\\\"\\\\\\\"\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(f\\\\\\\"\\\\\\\"\\\\\\\"\\\\n                        \\\\u003ch5\\\\u003e📋 Describe\\\\u003c/h5\\\\u003e\\\\n                        \\\\u003cpre\\\\u003e{matched_describe}\\\\u003c/pre\\\\u003e\\\\n                    \\\\\\\"\\\\\\\"\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(f\\\\\\\"\\\\\\\"\\\\\\\"\\\\n                        \\\\u003ch5\\\\u003e🧾 Logs\\\\u003c/h5\\\\u003e\\\\n                        \\\\u003cpre\\\\u003e{matched_logs}\\\\u003c/pre\\\\u003e\\\\n                    \\\\\\\"\\\\\\\"\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/div\\\\u003e\\\\u003c/div\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in events.splitlines() if keyword in l])\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\\\\"\\\\\\\"\\\\n            \\\\u003cdiv class=\\\\\\\"card border-warning\\\\\\\"\\\\u003e\\\\n                \\\\u003cdiv class=\\\\\\\"card-header bg-warning text-dark\\\\\\\"\\\\u003e\\\\n                    ⚠️ Events in Namespace: \\\\u003cstrong\\\\u003e{ns}\\\\u003c/strong\\\\u003e\\\\n                \\\\u003c/div\\\\u003e\\\\n                \\\\u003cdiv class=\\\\\\\"card-body\\\\\\\"\\\\u003e\\\\n                    \\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\n                \\\\u003c/div\\\\u003e\\\\n            \\\\u003c/div\\\\u003e\\\\n            \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        # HTML 끝\\\\n        f.write(\\\\\\\"\\\\u003c/div\\\\u003e\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"'  \\\\u0026\\\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str\\\\n):\\\\n\\\\n    import subprocess\\\\n    import shutil\\\\n    import os\\\\n    from datetime import datetime\\\\n\\\\n    os.environ[\\\\\\\"GIT_ASKPASS\\\\\\\"] = \\\\\\\"echo\\\\\\\"\\\\n\\\\n    report_path = report_file.path\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_path)\\\\n    if not os.path.exists(report_path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_path} does not exist\\\\\\\")\\\\n\\\\n    # HTML 파일 이름 만들기\\\\n    timestamp = datetime.now().strftime(\\\\\\\"report-%Y%m%d-%H%M%S.html\\\\\\\")\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n    report_target_path = os.path.join(repo_dir, timestamp)\\\\n\\\\n    # Git clone\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    shutil.copyfile(report_path, report_target_path)\\\\n\\\\n    # Git config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    # Git commit \\\\u0026 push\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", f\\\\\\\"Add {timestamp}\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(f\\\\\\\"Report committed and pushed to GitHub as {timestamp}.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Run of pod_log_pipeline_advance_version_at_2025-05-08T06:28:34.201Z (ee0f7)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:28:52Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}},{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:28:52Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"ecfd93e6-166e-4811-920a-fb70c4edd9f1\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-filtered-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-filtered-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-filtered-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-filtered-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-filtered-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-filtered-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"ecfd93e6-166e-4811-920a-fb70c4edd9f1\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"ecfd93e6-166e-4811-920a-fb70c4edd9f1\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:28:52Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"nodes\":{\"pod-log-to-github-jt27k\":{\"id\":\"pod-log-to-github-jt27k\",\"name\":\"pod-log-to-github-jt27k\",\"displayName\":\"pod-log-to-github-jt27k\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-jt27k\",\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:28:52Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"children\":[\"pod-log-to-github-jt27k-490722581\"]},\"pod-log-to-github-jt27k-490722581\":{\"id\":\"pod-log-to-github-jt27k-490722581\",\"name\":\"pod-log-to-github-jt27k.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-jt27k\",\"phase\":\"Pending\",\"boundaryID\":\"pod-log-to-github-jt27k\",\"startedAt\":\"2025-05-08T06:28:52Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]}}},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = NotFound desc = Failed to report workflow: Failed to report workflow name=\"pod-log-to-github-jt27k\" namespace=\"kubeflow-user-example-com\" runId=\"ecfd93e6-166e-4811-920a-fb70c4edd9f1\": Failed to fetch run ecfd93e6-166e-4811-920a-fb70c4edd9f1: ResourceNotFoundError: Run ecfd93e6-166e-4811-920a-fb70c4edd9f1 not found: CustomError (code: 1): Error while reporting workflow resource (code: NotFound, message: Failed to report workflow: Failed to report workflow name=\"pod-log-to-github-jt27k\" namespace=\"kubeflow-user-example-com\" runId=\"ecfd93e6-166e-4811-920a-fb70c4edd9f1\": Failed to fetch run ecfd93e6-166e-4811-920a-fb70c4edd9f1: ResourceNotFoundError: Run ecfd93e6-166e-4811-920a-fb70c4edd9f1 not found): rpc error: code = NotFound desc = Failed to report workflow: Failed to report workflow name=\"pod-log-to-github-jt27k\" namespace=\"kubeflow-user-example-com\" runId=\"ecfd93e6-166e-4811-920a-fb70c4edd9f1\": Failed to fetch run ecfd93e6-166e-4811-920a-fb70c4edd9f1: ResourceNotFoundError: Run ecfd93e6-166e-4811-920a-fb70c4edd9f1 not found, {\"metadata\":{\"name\":\"pod-log-to-github-jt27k\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"6320da99-118e-4c98-af55-42f7b61b864c\",\"resourceVersion\":\"6552414\",\"generation\":2,\"creationTimestamp\":\"2025-05-08T06:28:52Z\",\"labels\":{\"pipeline/runid\":\"ecfd93e6-166e-4811-920a-fb70c4edd9f1\",\"workflows.argoproj.io/phase\":\"Running\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-filtered-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"defaultValue\\\":\\\"crash\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"}}},\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\\\\\"crash\\\\\\\"):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Pod 목록 가져오기\\\\n    pods_output = subprocess.getoutput(\\\\n        \\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\"\\\\n    )\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML 시작\\\\n        f.write(f\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"utf-8\\\\\\\"\\\\u003e\\\\n            \\\\u003ctitle\\\\u003e장애 복기 리포트\\\\u003c/title\\\\u003e\\\\n            \\\\u003clink href=\\\\\\\"https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/css/bootstrap.min.css\\\\\\\" rel=\\\\\\\"stylesheet\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body {{ padding: 2rem; background-color: #f8f9fa; }}\\\\n                pre {{ background-color: #f1f1f1; padding: 1rem; border-radius: 5px; overflow-x: auto; }}\\\\n                .card {{ margin-bottom: 2rem; }}\\\\n                .keyword {{ background-color: yellow; font-weight: bold; }}\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n            \\\\u003cdiv class=\\\\\\\"container\\\\\\\"\\\\u003e\\\\n            \\\\u003ch1 class=\\\\\\\"mb-4\\\\\\\"\\\\u003e🚨 장애 복기 리포트\\\\u003c/h1\\\\u003e\\\\n            \\\\u003ch5 class=\\\\\\\"text-muted\\\\\\\"\\\\u003eKeyword: \\\\u003cspan class=\\\\\\\"badge bg-danger\\\\\\\"\\\\u003e{keyword}\\\\u003c/span\\\\u003e\\\\u003c/h5\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in describe.splitlines() if keyword in l])\\\\n            matched_logs = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in logs.splitlines() if keyword in l])\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\\\\"\\\\\\\"\\\\n                \\\\u003cdiv class=\\\\\\\"card\\\\\\\"\\\\u003e\\\\n                    \\\\u003cdiv class=\\\\\\\"card-header bg-primary text-white\\\\\\\"\\\\u003e\\\\n                        Pod: \\\\u003cstrong\\\\u003e{pod_name}\\\\u003c/strong\\\\u003e \\\\u0026nbsp;\\\\u0026nbsp;|\\\\u0026nbsp;\\\\u0026nbsp; Namespace: \\\\u003cstrong\\\\u003e{namespace}\\\\u003c/strong\\\\u003e\\\\n                    \\\\u003c/div\\\\u003e\\\\n                    \\\\u003cdiv class=\\\\\\\"card-body\\\\\\\"\\\\u003e\\\\n                \\\\\\\"\\\\\\\"\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(f\\\\\\\"\\\\\\\"\\\\\\\"\\\\n                        \\\\u003ch5\\\\u003e📋 Describe\\\\u003c/h5\\\\u003e\\\\n                        \\\\u003cpre\\\\u003e{matched_describe}\\\\u003c/pre\\\\u003e\\\\n                    \\\\\\\"\\\\\\\"\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(f\\\\\\\"\\\\\\\"\\\\\\\"\\\\n                        \\\\u003ch5\\\\u003e🧾 Logs\\\\u003c/h5\\\\u003e\\\\n                        \\\\u003cpre\\\\u003e{matched_logs}\\\\u003c/pre\\\\u003e\\\\n                    \\\\\\\"\\\\\\\"\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/div\\\\u003e\\\\u003c/div\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in events.splitlines() if keyword in l])\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\\\\"\\\\\\\"\\\\n            \\\\u003cdiv class=\\\\\\\"card border-warning\\\\\\\"\\\\u003e\\\\n                \\\\u003cdiv class=\\\\\\\"card-header bg-warning text-dark\\\\\\\"\\\\u003e\\\\n                    ⚠️ Events in Namespace: \\\\u003cstrong\\\\u003e{ns}\\\\u003c/strong\\\\u003e\\\\n                \\\\u003c/div\\\\u003e\\\\n                \\\\u003cdiv class=\\\\\\\"card-body\\\\\\\"\\\\u003e\\\\n                    \\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\n                \\\\u003c/div\\\\u003e\\\\n            \\\\u003c/div\\\\u003e\\\\n            \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        # HTML 끝\\\\n        f.write(\\\\\\\"\\\\u003c/div\\\\u003e\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"'  \\\\u0026\\\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str\\\\n):\\\\n\\\\n    import subprocess\\\\n    import shutil\\\\n    import os\\\\n    from datetime import datetime\\\\n\\\\n    os.environ[\\\\\\\"GIT_ASKPASS\\\\\\\"] = \\\\\\\"echo\\\\\\\"\\\\n\\\\n    report_path = report_file.path\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_path)\\\\n    if not os.path.exists(report_path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_path} does not exist\\\\\\\")\\\\n\\\\n    # HTML 파일 이름 만들기\\\\n    timestamp = datetime.now().strftime(\\\\\\\"report-%Y%m%d-%H%M%S.html\\\\\\\")\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n    report_target_path = os.path.join(repo_dir, timestamp)\\\\n\\\\n    # Git clone\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    shutil.copyfile(report_path, report_target_path)\\\\n\\\\n    # Git config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    # Git commit \\\\u0026 push\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", f\\\\\\\"Add {timestamp}\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(f\\\\\\\"Report committed and pushed to GitHub as {timestamp}.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Run of pod_log_pipeline_advance_version_at_2025-05-08T06:28:34.201Z (ee0f7)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:28:52Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}},{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:28:52Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"ecfd93e6-166e-4811-920a-fb70c4edd9f1\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-filtered-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-filtered-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-filtered-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-filtered-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-filtered-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-filtered-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"ecfd93e6-166e-4811-920a-fb70c4edd9f1\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"ecfd93e6-166e-4811-920a-fb70c4edd9f1\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:28:52Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"nodes\":{\"pod-log-to-github-jt27k\":{\"id\":\"pod-log-to-github-jt27k\",\"name\":\"pod-log-to-github-jt27k\",\"displayName\":\"pod-log-to-github-jt27k\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-jt27k\",\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:28:52Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"children\":[\"pod-log-to-github-jt27k-490722581\"]},\"pod-log-to-github-jt27k-490722581\":{\"id\":\"pod-log-to-github-jt27k-490722581\",\"name\":\"pod-log-to-github-jt27k.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-jt27k\",\"phase\":\"Pending\",\"boundaryID\":\"pod-log-to-github-jt27k\",\"startedAt\":\"2025-05-08T06:28:52Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]}}},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = NotFound desc = Failed to report workflow: Failed to report workflow name=\"pod-log-to-github-jt27k\" namespace=\"kubeflow-user-example-com\" runId=\"ecfd93e6-166e-4811-920a-fb70c4edd9f1\": Failed to fetch run ecfd93e6-166e-4811-920a-fb70c4edd9f1: ResourceNotFoundError: Run ecfd93e6-166e-4811-920a-fb70c4edd9f1 not found"
time="2025-05-08T06:28:53Z" level=error msg="Permanent failure while syncing resource (kubeflow-user-example-com/pod-log-to-github-jt27k): CustomError (code: 1): Workflow (kubeflow-user-example-com/pod-log-to-github-jt27k) in work queue no longer exists: CustomError (code: 2): Error retrieving workflow (pod-log-to-github-jt27k) in namespace (kubeflow-user-example-com): Error retrieving workflow (pod-log-to-github-jt27k) in namespace (kubeflow-user-example-com): workflow.argoproj.io \"pod-log-to-github-jt27k\" not found: workflow.argoproj.io \"pod-log-to-github-jt27k\" not found: Error retrieving workflow (pod-log-to-github-jt27k) in namespace (kubeflow-user-example-com): workflow.argoproj.io \"pod-log-to-github-jt27k\" not found: workflow.argoproj.io \"pod-log-to-github-jt27k\" not found: CustomError (code: 2): Error retrieving workflow (pod-log-to-github-jt27k) in namespace (kubeflow-user-example-com): Error retrieving workflow (pod-log-to-github-jt27k) in namespace (kubeflow-user-example-com): workflow.argoproj.io \"pod-log-to-github-jt27k\" not found: workflow.argoproj.io \"pod-log-to-github-jt27k\" not found: Error retrieving workflow (pod-log-to-github-jt27k) in namespace (kubeflow-user-example-com): workflow.argoproj.io \"pod-log-to-github-jt27k\" not found: workflow.argoproj.io \"pod-log-to-github-jt27k\" not found"
time="2025-05-08T06:28:58Z" level=error msg="Permanent failure while syncing resource (kubeflow-user-example-com/pod-log-to-github-kg92g): CustomError (code: 1): Syncing Workflow (pod-log-to-github-kg92g): permanent failure: CustomError (code: 1): Error while reporting workflow resource (code: NotFound, message: Failed to report workflow: Failed to report workflow name=\"pod-log-to-github-kg92g\" namespace=\"kubeflow-user-example-com\" runId=\"d1b8ce1d-db0c-401a-9d6a-4e8b08c51496\": Failed to fetch run d1b8ce1d-db0c-401a-9d6a-4e8b08c51496: ResourceNotFoundError: Run d1b8ce1d-db0c-401a-9d6a-4e8b08c51496 not found): rpc error: code = NotFound desc = Failed to report workflow: Failed to report workflow name=\"pod-log-to-github-kg92g\" namespace=\"kubeflow-user-example-com\" runId=\"d1b8ce1d-db0c-401a-9d6a-4e8b08c51496\": Failed to fetch run d1b8ce1d-db0c-401a-9d6a-4e8b08c51496: ResourceNotFoundError: Run d1b8ce1d-db0c-401a-9d6a-4e8b08c51496 not found, {\"metadata\":{\"name\":\"pod-log-to-github-kg92g\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"3935aaee-a404-459e-a014-5738d72c8592\",\"resourceVersion\":\"6552462\",\"generation\":2,\"creationTimestamp\":\"2025-05-08T06:28:57Z\",\"labels\":{\"pipeline/runid\":\"d1b8ce1d-db0c-401a-9d6a-4e8b08c51496\",\"workflows.argoproj.io/phase\":\"Running\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-filtered-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"defaultValue\\\":\\\"crash\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"}}},\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\\\\\"crash\\\\\\\"):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Pod 목록 가져오기\\\\n    pods_output = subprocess.getoutput(\\\\n        \\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\"\\\\n    )\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML 시작\\\\n        f.write(f\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"utf-8\\\\\\\"\\\\u003e\\\\n            \\\\u003ctitle\\\\u003e장애 복기 리포트\\\\u003c/title\\\\u003e\\\\n            \\\\u003clink href=\\\\\\\"https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/css/bootstrap.min.css\\\\\\\" rel=\\\\\\\"stylesheet\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body {{ padding: 2rem; background-color: #f8f9fa; }}\\\\n                pre {{ background-color: #f1f1f1; padding: 1rem; border-radius: 5px; overflow-x: auto; }}\\\\n                .card {{ margin-bottom: 2rem; }}\\\\n                .keyword {{ background-color: yellow; font-weight: bold; }}\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n            \\\\u003cdiv class=\\\\\\\"container\\\\\\\"\\\\u003e\\\\n            \\\\u003ch1 class=\\\\\\\"mb-4\\\\\\\"\\\\u003e🚨 장애 복기 리포트\\\\u003c/h1\\\\u003e\\\\n            \\\\u003ch5 class=\\\\\\\"text-muted\\\\\\\"\\\\u003eKeyword: \\\\u003cspan class=\\\\\\\"badge bg-danger\\\\\\\"\\\\u003e{keyword}\\\\u003c/span\\\\u003e\\\\u003c/h5\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in describe.splitlines() if keyword in l])\\\\n            matched_logs = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in logs.splitlines() if keyword in l])\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\\\\"\\\\\\\"\\\\n                \\\\u003cdiv class=\\\\\\\"card\\\\\\\"\\\\u003e\\\\n                    \\\\u003cdiv class=\\\\\\\"card-header bg-primary text-white\\\\\\\"\\\\u003e\\\\n                        Pod: \\\\u003cstrong\\\\u003e{pod_name}\\\\u003c/strong\\\\u003e \\\\u0026nbsp;\\\\u0026nbsp;|\\\\u0026nbsp;\\\\u0026nbsp; Namespace: \\\\u003cstrong\\\\u003e{namespace}\\\\u003c/strong\\\\u003e\\\\n                    \\\\u003c/div\\\\u003e\\\\n                    \\\\u003cdiv class=\\\\\\\"card-body\\\\\\\"\\\\u003e\\\\n                \\\\\\\"\\\\\\\"\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(f\\\\\\\"\\\\\\\"\\\\\\\"\\\\n                        \\\\u003ch5\\\\u003e📋 Describe\\\\u003c/h5\\\\u003e\\\\n                        \\\\u003cpre\\\\u003e{matched_describe}\\\\u003c/pre\\\\u003e\\\\n                    \\\\\\\"\\\\\\\"\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(f\\\\\\\"\\\\\\\"\\\\\\\"\\\\n                        \\\\u003ch5\\\\u003e🧾 Logs\\\\u003c/h5\\\\u003e\\\\n                        \\\\u003cpre\\\\u003e{matched_logs}\\\\u003c/pre\\\\u003e\\\\n                    \\\\\\\"\\\\\\\"\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/div\\\\u003e\\\\u003c/div\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in events.splitlines() if keyword in l])\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\\\\"\\\\\\\"\\\\n            \\\\u003cdiv class=\\\\\\\"card border-warning\\\\\\\"\\\\u003e\\\\n                \\\\u003cdiv class=\\\\\\\"card-header bg-warning text-dark\\\\\\\"\\\\u003e\\\\n                    ⚠️ Events in Namespace: \\\\u003cstrong\\\\u003e{ns}\\\\u003c/strong\\\\u003e\\\\n                \\\\u003c/div\\\\u003e\\\\n                \\\\u003cdiv class=\\\\\\\"card-body\\\\\\\"\\\\u003e\\\\n                    \\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\n                \\\\u003c/div\\\\u003e\\\\n            \\\\u003c/div\\\\u003e\\\\n            \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        # HTML 끝\\\\n        f.write(\\\\\\\"\\\\u003c/div\\\\u003e\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"'  \\\\u0026\\\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str\\\\n):\\\\n\\\\n    import subprocess\\\\n    import shutil\\\\n    import os\\\\n    from datetime import datetime\\\\n\\\\n    os.environ[\\\\\\\"GIT_ASKPASS\\\\\\\"] = \\\\\\\"echo\\\\\\\"\\\\n\\\\n    report_path = report_file.path\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_path)\\\\n    if not os.path.exists(report_path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_path} does not exist\\\\\\\")\\\\n\\\\n    # HTML 파일 이름 만들기\\\\n    timestamp = datetime.now().strftime(\\\\\\\"report-%Y%m%d-%H%M%S.html\\\\\\\")\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n    report_target_path = os.path.join(repo_dir, timestamp)\\\\n\\\\n    # Git clone\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    shutil.copyfile(report_path, report_target_path)\\\\n\\\\n    # Git config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    # Git commit \\\\u0026 push\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", f\\\\\\\"Add {timestamp}\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(f\\\\\\\"Report committed and pushed to GitHub as {timestamp}.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Run of pod_log_pipeline_advance_version_at_2025-05-08T06:28:34.201Z (ee0f7)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:28:57Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}},{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:28:57Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"d1b8ce1d-db0c-401a-9d6a-4e8b08c51496\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-filtered-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-filtered-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-filtered-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-filtered-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-filtered-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-filtered-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"d1b8ce1d-db0c-401a-9d6a-4e8b08c51496\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"d1b8ce1d-db0c-401a-9d6a-4e8b08c51496\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:28:57Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"nodes\":{\"pod-log-to-github-kg92g\":{\"id\":\"pod-log-to-github-kg92g\",\"name\":\"pod-log-to-github-kg92g\",\"displayName\":\"pod-log-to-github-kg92g\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-kg92g\",\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:28:57Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"children\":[\"pod-log-to-github-kg92g-659157945\"]},\"pod-log-to-github-kg92g-659157945\":{\"id\":\"pod-log-to-github-kg92g-659157945\",\"name\":\"pod-log-to-github-kg92g.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-kg92g\",\"phase\":\"Pending\",\"boundaryID\":\"pod-log-to-github-kg92g\",\"startedAt\":\"2025-05-08T06:28:57Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]}}},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = NotFound desc = Failed to report workflow: Failed to report workflow name=\"pod-log-to-github-kg92g\" namespace=\"kubeflow-user-example-com\" runId=\"d1b8ce1d-db0c-401a-9d6a-4e8b08c51496\": Failed to fetch run d1b8ce1d-db0c-401a-9d6a-4e8b08c51496: ResourceNotFoundError: Run d1b8ce1d-db0c-401a-9d6a-4e8b08c51496 not found: CustomError (code: 1): Error while reporting workflow resource (code: NotFound, message: Failed to report workflow: Failed to report workflow name=\"pod-log-to-github-kg92g\" namespace=\"kubeflow-user-example-com\" runId=\"d1b8ce1d-db0c-401a-9d6a-4e8b08c51496\": Failed to fetch run d1b8ce1d-db0c-401a-9d6a-4e8b08c51496: ResourceNotFoundError: Run d1b8ce1d-db0c-401a-9d6a-4e8b08c51496 not found): rpc error: code = NotFound desc = Failed to report workflow: Failed to report workflow name=\"pod-log-to-github-kg92g\" namespace=\"kubeflow-user-example-com\" runId=\"d1b8ce1d-db0c-401a-9d6a-4e8b08c51496\": Failed to fetch run d1b8ce1d-db0c-401a-9d6a-4e8b08c51496: ResourceNotFoundError: Run d1b8ce1d-db0c-401a-9d6a-4e8b08c51496 not found, {\"metadata\":{\"name\":\"pod-log-to-github-kg92g\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"3935aaee-a404-459e-a014-5738d72c8592\",\"resourceVersion\":\"6552462\",\"generation\":2,\"creationTimestamp\":\"2025-05-08T06:28:57Z\",\"labels\":{\"pipeline/runid\":\"d1b8ce1d-db0c-401a-9d6a-4e8b08c51496\",\"workflows.argoproj.io/phase\":\"Running\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-filtered-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"keyword\\\":{\\\"defaultValue\\\":\\\"crash\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"}}},\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset], keyword: str = \\\\\\\"crash\\\\\\\"):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Pod 목록 가져오기\\\\n    pods_output = subprocess.getoutput(\\\\n        \\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\"\\\\n    )\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML 시작\\\\n        f.write(f\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"utf-8\\\\\\\"\\\\u003e\\\\n            \\\\u003ctitle\\\\u003e장애 복기 리포트\\\\u003c/title\\\\u003e\\\\n            \\\\u003clink href=\\\\\\\"https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/css/bootstrap.min.css\\\\\\\" rel=\\\\\\\"stylesheet\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body {{ padding: 2rem; background-color: #f8f9fa; }}\\\\n                pre {{ background-color: #f1f1f1; padding: 1rem; border-radius: 5px; overflow-x: auto; }}\\\\n                .card {{ margin-bottom: 2rem; }}\\\\n                .keyword {{ background-color: yellow; font-weight: bold; }}\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n            \\\\u003cdiv class=\\\\\\\"container\\\\\\\"\\\\u003e\\\\n            \\\\u003ch1 class=\\\\\\\"mb-4\\\\\\\"\\\\u003e🚨 장애 복기 리포트\\\\u003c/h1\\\\u003e\\\\n            \\\\u003ch5 class=\\\\\\\"text-muted\\\\\\\"\\\\u003eKeyword: \\\\u003cspan class=\\\\\\\"badge bg-danger\\\\\\\"\\\\u003e{keyword}\\\\u003c/span\\\\u003e\\\\u003c/h5\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            logs = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in describe.splitlines() if keyword in l])\\\\n            matched_logs = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in logs.splitlines() if keyword in l])\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\\\\"\\\\\\\"\\\\n                \\\\u003cdiv class=\\\\\\\"card\\\\\\\"\\\\u003e\\\\n                    \\\\u003cdiv class=\\\\\\\"card-header bg-primary text-white\\\\\\\"\\\\u003e\\\\n                        Pod: \\\\u003cstrong\\\\u003e{pod_name}\\\\u003c/strong\\\\u003e \\\\u0026nbsp;\\\\u0026nbsp;|\\\\u0026nbsp;\\\\u0026nbsp; Namespace: \\\\u003cstrong\\\\u003e{namespace}\\\\u003c/strong\\\\u003e\\\\n                    \\\\u003c/div\\\\u003e\\\\n                    \\\\u003cdiv class=\\\\\\\"card-body\\\\\\\"\\\\u003e\\\\n                \\\\\\\"\\\\\\\"\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(f\\\\\\\"\\\\\\\"\\\\\\\"\\\\n                        \\\\u003ch5\\\\u003e📋 Describe\\\\u003c/h5\\\\u003e\\\\n                        \\\\u003cpre\\\\u003e{matched_describe}\\\\u003c/pre\\\\u003e\\\\n                    \\\\\\\"\\\\\\\"\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(f\\\\\\\"\\\\\\\"\\\\\\\"\\\\n                        \\\\u003ch5\\\\u003e🧾 Logs\\\\u003c/h5\\\\u003e\\\\n                        \\\\u003cpre\\\\u003e{matched_logs}\\\\u003c/pre\\\\u003e\\\\n                    \\\\\\\"\\\\\\\"\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/div\\\\u003e\\\\u003c/div\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = \\\\\\\"\\\\\\\\n\\\\\\\".join([l for l in events.splitlines() if keyword in l])\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\\\\"\\\\\\\"\\\\n            \\\\u003cdiv class=\\\\\\\"card border-warning\\\\\\\"\\\\u003e\\\\n                \\\\u003cdiv class=\\\\\\\"card-header bg-warning text-dark\\\\\\\"\\\\u003e\\\\n                    ⚠️ Events in Namespace: \\\\u003cstrong\\\\u003e{ns}\\\\u003c/strong\\\\u003e\\\\n                \\\\u003c/div\\\\u003e\\\\n                \\\\u003cdiv class=\\\\\\\"card-body\\\\\\\"\\\\u003e\\\\n                    \\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\n                \\\\u003c/div\\\\u003e\\\\n            \\\\u003c/div\\\\u003e\\\\n            \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        # HTML 끝\\\\n        f.write(\\\\\\\"\\\\u003c/div\\\\u003e\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"'  \\\\u0026\\\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str\\\\n):\\\\n\\\\n    import subprocess\\\\n    import shutil\\\\n    import os\\\\n    from datetime import datetime\\\\n\\\\n    os.environ[\\\\\\\"GIT_ASKPASS\\\\\\\"] = \\\\\\\"echo\\\\\\\"\\\\n\\\\n    report_path = report_file.path\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_path)\\\\n    if not os.path.exists(report_path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_path} does not exist\\\\\\\")\\\\n\\\\n    # HTML 파일 이름 만들기\\\\n    timestamp = datetime.now().strftime(\\\\\\\"report-%Y%m%d-%H%M%S.html\\\\\\\")\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n    report_target_path = os.path.join(repo_dir, timestamp)\\\\n\\\\n    # Git clone\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    shutil.copyfile(report_path, report_target_path)\\\\n\\\\n    # Git config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    # Git commit \\\\u0026 push\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", f\\\\\\\"Add {timestamp}\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(f\\\\\\\"Report committed and pushed to GitHub as {timestamp}.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Run of pod_log_pipeline_advance_version_at_2025-05-08T06:28:34.201Z (ee0f7)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:28:57Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}},{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:28:57Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"d1b8ce1d-db0c-401a-9d6a-4e8b08c51496\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-filtered-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-filtered-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-filtered-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-filtered-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-filtered-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-filtered-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"d1b8ce1d-db0c-401a-9d6a-4e8b08c51496\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"d1b8ce1d-db0c-401a-9d6a-4e8b08c51496\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:28:57Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"nodes\":{\"pod-log-to-github-kg92g\":{\"id\":\"pod-log-to-github-kg92g\",\"name\":\"pod-log-to-github-kg92g\",\"displayName\":\"pod-log-to-github-kg92g\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-kg92g\",\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:28:57Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"children\":[\"pod-log-to-github-kg92g-659157945\"]},\"pod-log-to-github-kg92g-659157945\":{\"id\":\"pod-log-to-github-kg92g-659157945\",\"name\":\"pod-log-to-github-kg92g.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-kg92g\",\"phase\":\"Pending\",\"boundaryID\":\"pod-log-to-github-kg92g\",\"startedAt\":\"2025-05-08T06:28:57Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]}}},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = NotFound desc = Failed to report workflow: Failed to report workflow name=\"pod-log-to-github-kg92g\" namespace=\"kubeflow-user-example-com\" runId=\"d1b8ce1d-db0c-401a-9d6a-4e8b08c51496\": Failed to fetch run d1b8ce1d-db0c-401a-9d6a-4e8b08c51496: ResourceNotFoundError: Run d1b8ce1d-db0c-401a-9d6a-4e8b08c51496 not found"
time="2025-05-08T06:28:58Z" level=error msg="Permanent failure while syncing resource (kubeflow-user-example-com/pod-log-to-github-kg92g): CustomError (code: 1): Workflow (kubeflow-user-example-com/pod-log-to-github-kg92g) in work queue no longer exists: CustomError (code: 2): Error retrieving workflow (pod-log-to-github-kg92g) in namespace (kubeflow-user-example-com): Error retrieving workflow (pod-log-to-github-kg92g) in namespace (kubeflow-user-example-com): workflow.argoproj.io \"pod-log-to-github-kg92g\" not found: workflow.argoproj.io \"pod-log-to-github-kg92g\" not found: Error retrieving workflow (pod-log-to-github-kg92g) in namespace (kubeflow-user-example-com): workflow.argoproj.io \"pod-log-to-github-kg92g\" not found: workflow.argoproj.io \"pod-log-to-github-kg92g\" not found: CustomError (code: 2): Error retrieving workflow (pod-log-to-github-kg92g) in namespace (kubeflow-user-example-com): Error retrieving workflow (pod-log-to-github-kg92g) in namespace (kubeflow-user-example-com): workflow.argoproj.io \"pod-log-to-github-kg92g\" not found: workflow.argoproj.io \"pod-log-to-github-kg92g\" not found: Error retrieving workflow (pod-log-to-github-kg92g) in namespace (kubeflow-user-example-com): workflow.argoproj.io \"pod-log-to-github-kg92g\" not found: workflow.argoproj.io \"pod-log-to-github-kg92g\" not found"
time="2025-05-08T06:31:43Z" level=error msg="Transient failure while syncing resource (kubeflow-user-example-com/pod-log-to-github-z2xx6): CustomError (code: 0): Syncing Workflow (pod-log-to-github-z2xx6): transient failure: CustomError (code: 0): Error while reporting workflow resource (code: Unauthenticated, message: Jwt is expired): rpc error: code = Unauthenticated desc = Jwt is expired, {\"metadata\":{\"name\":\"pod-log-to-github-z2xx6\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"cc5e663d-85d9-4747-9066-6673e479fa1b\",\"resourceVersion\":\"6553428\",\"generation\":2,\"creationTimestamp\":\"2025-05-08T06:31:42Z\",\"labels\":{\"pipeline/runid\":\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"workflows.argoproj.io/phase\":\"Running\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-filtered-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"'  \\\\u0026\\\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str\\\\n):\\\\n\\\\n    import subprocess\\\\n    import shutil\\\\n    import os\\\\n    from datetime import datetime\\\\n\\\\n    os.environ[\\\\\\\"GIT_ASKPASS\\\\\\\"] = \\\\\\\"echo\\\\\\\"\\\\n\\\\n    report_path = report_file.path\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_path)\\\\n    if not os.path.exists(report_path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_path} does not exist\\\\\\\")\\\\n\\\\n    # HTML 파일 이름 만들기\\\\n    timestamp = datetime.now().strftime(\\\\\\\"report-%Y%m%d-%H%M%S.html\\\\\\\")\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n    report_target_path = os.path.join(repo_dir, timestamp)\\\\n\\\\n    # Git clone\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    shutil.copyfile(report_path, report_target_path)\\\\n\\\\n    # Git config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    # Git commit \\\\u0026 push\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", f\\\\\\\"Add {timestamp}\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(f\\\\\\\"Report committed and pushed to GitHub as {timestamp}.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Run of pod_log_pipeline_advance_version_at_2025-05-08T06:31:24.503Z (1347f)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:31:42Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}},{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:31:42Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-filtered-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-filtered-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-filtered-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-filtered-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-filtered-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-filtered-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:31:42Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"nodes\":{\"pod-log-to-github-z2xx6\":{\"id\":\"pod-log-to-github-z2xx6\",\"name\":\"pod-log-to-github-z2xx6\",\"displayName\":\"pod-log-to-github-z2xx6\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-z2xx6\",\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:31:42Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"children\":[\"pod-log-to-github-z2xx6-1009858533\"]},\"pod-log-to-github-z2xx6-1009858533\":{\"id\":\"pod-log-to-github-z2xx6-1009858533\",\"name\":\"pod-log-to-github-z2xx6.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-z2xx6\",\"phase\":\"Pending\",\"boundaryID\":\"pod-log-to-github-z2xx6\",\"startedAt\":\"2025-05-08T06:31:42Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]}}},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = Unauthenticated desc = Jwt is expired: CustomError (code: 0): Error while reporting workflow resource (code: Unauthenticated, message: Jwt is expired): rpc error: code = Unauthenticated desc = Jwt is expired, {\"metadata\":{\"name\":\"pod-log-to-github-z2xx6\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"cc5e663d-85d9-4747-9066-6673e479fa1b\",\"resourceVersion\":\"6553428\",\"generation\":2,\"creationTimestamp\":\"2025-05-08T06:31:42Z\",\"labels\":{\"pipeline/runid\":\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"workflows.argoproj.io/phase\":\"Running\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-filtered-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"'  \\\\u0026\\\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str\\\\n):\\\\n\\\\n    import subprocess\\\\n    import shutil\\\\n    import os\\\\n    from datetime import datetime\\\\n\\\\n    os.environ[\\\\\\\"GIT_ASKPASS\\\\\\\"] = \\\\\\\"echo\\\\\\\"\\\\n\\\\n    report_path = report_file.path\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_path)\\\\n    if not os.path.exists(report_path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_path} does not exist\\\\\\\")\\\\n\\\\n    # HTML 파일 이름 만들기\\\\n    timestamp = datetime.now().strftime(\\\\\\\"report-%Y%m%d-%H%M%S.html\\\\\\\")\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n    report_target_path = os.path.join(repo_dir, timestamp)\\\\n\\\\n    # Git clone\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    shutil.copyfile(report_path, report_target_path)\\\\n\\\\n    # Git config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    # Git commit \\\\u0026 push\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", f\\\\\\\"Add {timestamp}\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(f\\\\\\\"Report committed and pushed to GitHub as {timestamp}.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Run of pod_log_pipeline_advance_version_at_2025-05-08T06:31:24.503Z (1347f)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:31:42Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}},{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:31:42Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-filtered-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-filtered-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-filtered-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-filtered-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-filtered-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-filtered-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:31:42Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"nodes\":{\"pod-log-to-github-z2xx6\":{\"id\":\"pod-log-to-github-z2xx6\",\"name\":\"pod-log-to-github-z2xx6\",\"displayName\":\"pod-log-to-github-z2xx6\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-z2xx6\",\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:31:42Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"children\":[\"pod-log-to-github-z2xx6-1009858533\"]},\"pod-log-to-github-z2xx6-1009858533\":{\"id\":\"pod-log-to-github-z2xx6-1009858533\",\"name\":\"pod-log-to-github-z2xx6.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-z2xx6\",\"phase\":\"Pending\",\"boundaryID\":\"pod-log-to-github-z2xx6\",\"startedAt\":\"2025-05-08T06:31:42Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]}}},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = Unauthenticated desc = Jwt is expired"
time="2025-05-08T06:31:47Z" level=error msg="Transient failure while syncing resource (kubeflow-user-example-com/pod-log-to-github-z2xx6): CustomError (code: 0): Syncing Workflow (pod-log-to-github-z2xx6): transient failure: CustomError (code: 0): Error while reporting workflow resource (code: Unauthenticated, message: Jwt is expired): rpc error: code = Unauthenticated desc = Jwt is expired, {\"metadata\":{\"name\":\"pod-log-to-github-z2xx6\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"cc5e663d-85d9-4747-9066-6673e479fa1b\",\"resourceVersion\":\"6553428\",\"generation\":2,\"creationTimestamp\":\"2025-05-08T06:31:42Z\",\"labels\":{\"pipeline/runid\":\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"workflows.argoproj.io/phase\":\"Running\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-filtered-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"'  \\\\u0026\\\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str\\\\n):\\\\n\\\\n    import subprocess\\\\n    import shutil\\\\n    import os\\\\n    from datetime import datetime\\\\n\\\\n    os.environ[\\\\\\\"GIT_ASKPASS\\\\\\\"] = \\\\\\\"echo\\\\\\\"\\\\n\\\\n    report_path = report_file.path\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_path)\\\\n    if not os.path.exists(report_path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_path} does not exist\\\\\\\")\\\\n\\\\n    # HTML 파일 이름 만들기\\\\n    timestamp = datetime.now().strftime(\\\\\\\"report-%Y%m%d-%H%M%S.html\\\\\\\")\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n    report_target_path = os.path.join(repo_dir, timestamp)\\\\n\\\\n    # Git clone\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    shutil.copyfile(report_path, report_target_path)\\\\n\\\\n    # Git config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    # Git commit \\\\u0026 push\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", f\\\\\\\"Add {timestamp}\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(f\\\\\\\"Report committed and pushed to GitHub as {timestamp}.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Run of pod_log_pipeline_advance_version_at_2025-05-08T06:31:24.503Z (1347f)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:31:42Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}},{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:31:42Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-filtered-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-filtered-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-filtered-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-filtered-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-filtered-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-filtered-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:31:42Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"nodes\":{\"pod-log-to-github-z2xx6\":{\"id\":\"pod-log-to-github-z2xx6\",\"name\":\"pod-log-to-github-z2xx6\",\"displayName\":\"pod-log-to-github-z2xx6\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-z2xx6\",\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:31:42Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"children\":[\"pod-log-to-github-z2xx6-1009858533\"]},\"pod-log-to-github-z2xx6-1009858533\":{\"id\":\"pod-log-to-github-z2xx6-1009858533\",\"name\":\"pod-log-to-github-z2xx6.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-z2xx6\",\"phase\":\"Pending\",\"boundaryID\":\"pod-log-to-github-z2xx6\",\"startedAt\":\"2025-05-08T06:31:42Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]}}},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = Unauthenticated desc = Jwt is expired: CustomError (code: 0): Error while reporting workflow resource (code: Unauthenticated, message: Jwt is expired): rpc error: code = Unauthenticated desc = Jwt is expired, {\"metadata\":{\"name\":\"pod-log-to-github-z2xx6\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"cc5e663d-85d9-4747-9066-6673e479fa1b\",\"resourceVersion\":\"6553428\",\"generation\":2,\"creationTimestamp\":\"2025-05-08T06:31:42Z\",\"labels\":{\"pipeline/runid\":\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"workflows.argoproj.io/phase\":\"Running\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-filtered-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"'  \\\\u0026\\\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str\\\\n):\\\\n\\\\n    import subprocess\\\\n    import shutil\\\\n    import os\\\\n    from datetime import datetime\\\\n\\\\n    os.environ[\\\\\\\"GIT_ASKPASS\\\\\\\"] = \\\\\\\"echo\\\\\\\"\\\\n\\\\n    report_path = report_file.path\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_path)\\\\n    if not os.path.exists(report_path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_path} does not exist\\\\\\\")\\\\n\\\\n    # HTML 파일 이름 만들기\\\\n    timestamp = datetime.now().strftime(\\\\\\\"report-%Y%m%d-%H%M%S.html\\\\\\\")\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n    report_target_path = os.path.join(repo_dir, timestamp)\\\\n\\\\n    # Git clone\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    shutil.copyfile(report_path, report_target_path)\\\\n\\\\n    # Git config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    # Git commit \\\\u0026 push\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", f\\\\\\\"Add {timestamp}\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(f\\\\\\\"Report committed and pushed to GitHub as {timestamp}.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Run of pod_log_pipeline_advance_version_at_2025-05-08T06:31:24.503Z (1347f)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:31:42Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}},{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:31:42Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-filtered-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-filtered-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-filtered-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-filtered-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-filtered-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-filtered-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:31:42Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"nodes\":{\"pod-log-to-github-z2xx6\":{\"id\":\"pod-log-to-github-z2xx6\",\"name\":\"pod-log-to-github-z2xx6\",\"displayName\":\"pod-log-to-github-z2xx6\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-z2xx6\",\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:31:42Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"children\":[\"pod-log-to-github-z2xx6-1009858533\"]},\"pod-log-to-github-z2xx6-1009858533\":{\"id\":\"pod-log-to-github-z2xx6-1009858533\",\"name\":\"pod-log-to-github-z2xx6.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-z2xx6\",\"phase\":\"Pending\",\"boundaryID\":\"pod-log-to-github-z2xx6\",\"startedAt\":\"2025-05-08T06:31:42Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]}}},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = Unauthenticated desc = Jwt is expired"
time="2025-05-08T06:31:55Z" level=error msg="Transient failure while syncing resource (kubeflow-user-example-com/pod-log-to-github-z2xx6): CustomError (code: 0): Syncing Workflow (pod-log-to-github-z2xx6): transient failure: CustomError (code: 0): Error while reporting workflow resource (code: Unauthenticated, message: Jwt is expired): rpc error: code = Unauthenticated desc = Jwt is expired, {\"metadata\":{\"name\":\"pod-log-to-github-z2xx6\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"cc5e663d-85d9-4747-9066-6673e479fa1b\",\"resourceVersion\":\"6553502\",\"generation\":3,\"creationTimestamp\":\"2025-05-08T06:31:42Z\",\"labels\":{\"pipeline/runid\":\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"workflows.argoproj.io/phase\":\"Running\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-filtered-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"'  \\\\u0026\\\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str\\\\n):\\\\n\\\\n    import subprocess\\\\n    import shutil\\\\n    import os\\\\n    from datetime import datetime\\\\n\\\\n    os.environ[\\\\\\\"GIT_ASKPASS\\\\\\\"] = \\\\\\\"echo\\\\\\\"\\\\n\\\\n    report_path = report_file.path\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_path)\\\\n    if not os.path.exists(report_path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_path} does not exist\\\\\\\")\\\\n\\\\n    # HTML 파일 이름 만들기\\\\n    timestamp = datetime.now().strftime(\\\\\\\"report-%Y%m%d-%H%M%S.html\\\\\\\")\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n    report_target_path = os.path.join(repo_dir, timestamp)\\\\n\\\\n    # Git clone\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    shutil.copyfile(report_path, report_target_path)\\\\n\\\\n    # Git config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    # Git commit \\\\u0026 push\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", f\\\\\\\"Add {timestamp}\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(f\\\\\\\"Report committed and pushed to GitHub as {timestamp}.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Run of pod_log_pipeline_advance_version_at_2025-05-08T06:31:24.503Z (1347f)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:31:42Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}},{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:31:52Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-filtered-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-filtered-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-filtered-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-filtered-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-filtered-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-filtered-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:31:42Z\",\"finishedAt\":null,\"progress\":\"1/2\",\"nodes\":{\"pod-log-to-github-z2xx6\":{\"id\":\"pod-log-to-github-z2xx6\",\"name\":\"pod-log-to-github-z2xx6\",\"displayName\":\"pod-log-to-github-z2xx6\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-z2xx6\",\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:31:42Z\",\"finishedAt\":null,\"progress\":\"1/2\",\"children\":[\"pod-log-to-github-z2xx6-1009858533\"]},\"pod-log-to-github-z2xx6-1009858533\":{\"id\":\"pod-log-to-github-z2xx6-1009858533\",\"name\":\"pod-log-to-github-z2xx6.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-z2xx6\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-z2xx6\",\"startedAt\":\"2025-05-08T06:31:42Z\",\"finishedAt\":\"2025-05-08T06:31:46Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":4,\"memory\":3},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"value\":\"147\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"value\":\"0\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-z2xx6/2025/05/08/pod-log-to-github-z2xx6-system-dag-driver-1009858533/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-z2xx6-2144148312\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-z2xx6-2144148312\":{\"id\":\"pod-log-to-github-z2xx6-2144148312\",\"name\":\"pod-log-to-github-z2xx6.root\",\"displayName\":\"root\",\"type\":\"DAG\",\"templateName\":\"root\",\"templateScope\":\"local/pod-log-to-github-z2xx6\",\"phase\":\"Running\",\"boundaryID\":\"pod-log-to-github-z2xx6\",\"startedAt\":\"2025-05-08T06:31:52Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"147\"}]},\"children\":[\"pod-log-to-github-z2xx6-231423181\"]},\"pod-log-to-github-z2xx6-231423181\":{\"id\":\"pod-log-to-github-z2xx6-231423181\",\"name\":\"pod-log-to-github-z2xx6.root.collect-filtered-logs-driver\",\"displayName\":\"collect-filtered-logs-driver\",\"type\":\"Pod\",\"templateName\":\"system-container-driver\",\"templateScope\":\"local/pod-log-to-github-z2xx6\",\"phase\":\"Pending\",\"boundaryID\":\"pod-log-to-github-z2xx6-2144148312\",\"startedAt\":\"2025-05-08T06:31:52Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\"},{\"name\":\"parent-dag-id\",\"value\":\"147\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\",\"value\":\"\"}]}}},\"conditions\":[{\"type\":\"PodRunning\",\"status\":\"False\"}],\"resourcesDuration\":{\"cpu\":4,\"memory\":3},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = Unauthenticated desc = Jwt is expired: CustomError (code: 0): Error while reporting workflow resource (code: Unauthenticated, message: Jwt is expired): rpc error: code = Unauthenticated desc = Jwt is expired, {\"metadata\":{\"name\":\"pod-log-to-github-z2xx6\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"cc5e663d-85d9-4747-9066-6673e479fa1b\",\"resourceVersion\":\"6553502\",\"generation\":3,\"creationTimestamp\":\"2025-05-08T06:31:42Z\",\"labels\":{\"pipeline/runid\":\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"workflows.argoproj.io/phase\":\"Running\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-filtered-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"'  \\\\u0026\\\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str\\\\n):\\\\n\\\\n    import subprocess\\\\n    import shutil\\\\n    import os\\\\n    from datetime import datetime\\\\n\\\\n    os.environ[\\\\\\\"GIT_ASKPASS\\\\\\\"] = \\\\\\\"echo\\\\\\\"\\\\n\\\\n    report_path = report_file.path\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_path)\\\\n    if not os.path.exists(report_path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_path} does not exist\\\\\\\")\\\\n\\\\n    # HTML 파일 이름 만들기\\\\n    timestamp = datetime.now().strftime(\\\\\\\"report-%Y%m%d-%H%M%S.html\\\\\\\")\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n    report_target_path = os.path.join(repo_dir, timestamp)\\\\n\\\\n    # Git clone\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    shutil.copyfile(report_path, report_target_path)\\\\n\\\\n    # Git config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    # Git commit \\\\u0026 push\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", f\\\\\\\"Add {timestamp}\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(f\\\\\\\"Report committed and pushed to GitHub as {timestamp}.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Run of pod_log_pipeline_advance_version_at_2025-05-08T06:31:24.503Z (1347f)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:31:42Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}},{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:31:52Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-filtered-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-filtered-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-filtered-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-filtered-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-filtered-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-filtered-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:31:42Z\",\"finishedAt\":null,\"progress\":\"1/2\",\"nodes\":{\"pod-log-to-github-z2xx6\":{\"id\":\"pod-log-to-github-z2xx6\",\"name\":\"pod-log-to-github-z2xx6\",\"displayName\":\"pod-log-to-github-z2xx6\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-z2xx6\",\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:31:42Z\",\"finishedAt\":null,\"progress\":\"1/2\",\"children\":[\"pod-log-to-github-z2xx6-1009858533\"]},\"pod-log-to-github-z2xx6-1009858533\":{\"id\":\"pod-log-to-github-z2xx6-1009858533\",\"name\":\"pod-log-to-github-z2xx6.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-z2xx6\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-z2xx6\",\"startedAt\":\"2025-05-08T06:31:42Z\",\"finishedAt\":\"2025-05-08T06:31:46Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":4,\"memory\":3},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"value\":\"147\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"value\":\"0\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-z2xx6/2025/05/08/pod-log-to-github-z2xx6-system-dag-driver-1009858533/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-z2xx6-2144148312\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-z2xx6-2144148312\":{\"id\":\"pod-log-to-github-z2xx6-2144148312\",\"name\":\"pod-log-to-github-z2xx6.root\",\"displayName\":\"root\",\"type\":\"DAG\",\"templateName\":\"root\",\"templateScope\":\"local/pod-log-to-github-z2xx6\",\"phase\":\"Running\",\"boundaryID\":\"pod-log-to-github-z2xx6\",\"startedAt\":\"2025-05-08T06:31:52Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"147\"}]},\"children\":[\"pod-log-to-github-z2xx6-231423181\"]},\"pod-log-to-github-z2xx6-231423181\":{\"id\":\"pod-log-to-github-z2xx6-231423181\",\"name\":\"pod-log-to-github-z2xx6.root.collect-filtered-logs-driver\",\"displayName\":\"collect-filtered-logs-driver\",\"type\":\"Pod\",\"templateName\":\"system-container-driver\",\"templateScope\":\"local/pod-log-to-github-z2xx6\",\"phase\":\"Pending\",\"boundaryID\":\"pod-log-to-github-z2xx6-2144148312\",\"startedAt\":\"2025-05-08T06:31:52Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\"},{\"name\":\"parent-dag-id\",\"value\":\"147\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\",\"value\":\"\"}]}}},\"conditions\":[{\"type\":\"PodRunning\",\"status\":\"False\"}],\"resourcesDuration\":{\"cpu\":4,\"memory\":3},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = Unauthenticated desc = Jwt is expired"
time="2025-05-08T06:32:59Z" level=error msg="Transient failure while syncing resource (kubeflow-user-example-com/pod-log-to-github-z2xx6): CustomError (code: 0): Syncing Workflow (pod-log-to-github-z2xx6): transient failure: CustomError (code: 0): Error while reporting workflow resource (code: Unauthenticated, message: Jwt is expired): rpc error: code = Unauthenticated desc = Jwt is expired, {\"metadata\":{\"name\":\"pod-log-to-github-z2xx6\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"cc5e663d-85d9-4747-9066-6673e479fa1b\",\"resourceVersion\":\"6553659\",\"generation\":5,\"creationTimestamp\":\"2025-05-08T06:31:42Z\",\"labels\":{\"pipeline/runid\":\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"workflows.argoproj.io/phase\":\"Running\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-filtered-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"'  \\\\u0026\\\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str\\\\n):\\\\n\\\\n    import subprocess\\\\n    import shutil\\\\n    import os\\\\n    from datetime import datetime\\\\n\\\\n    os.environ[\\\\\\\"GIT_ASKPASS\\\\\\\"] = \\\\\\\"echo\\\\\\\"\\\\n\\\\n    report_path = report_file.path\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_path)\\\\n    if not os.path.exists(report_path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_path} does not exist\\\\\\\")\\\\n\\\\n    # HTML 파일 이름 만들기\\\\n    timestamp = datetime.now().strftime(\\\\\\\"report-%Y%m%d-%H%M%S.html\\\\\\\")\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n    report_target_path = os.path.join(repo_dir, timestamp)\\\\n\\\\n    # Git clone\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    shutil.copyfile(report_path, report_target_path)\\\\n\\\\n    # Git config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    # Git commit \\\\u0026 push\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", f\\\\\\\"Add {timestamp}\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(f\\\\\\\"Report committed and pushed to GitHub as {timestamp}.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Run of pod_log_pipeline_advance_version_at_2025-05-08T06:31:24.503Z (1347f)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:31:42Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}},{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:32:12Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-filtered-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-filtered-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-filtered-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-filtered-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-filtered-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-filtered-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:31:42Z\",\"finishedAt\":null,\"progress\":\"2/3\",\"nodes\":{\"pod-log-to-github-z2xx6\":{\"id\":\"pod-log-to-github-z2xx6\",\"name\":\"pod-log-to-github-z2xx6\",\"displayName\":\"pod-log-to-github-z2xx6\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-z2xx6\",\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:31:42Z\",\"finishedAt\":null,\"progress\":\"2/3\",\"children\":[\"pod-log-to-github-z2xx6-1009858533\"]},\"pod-log-to-github-z2xx6-1009858533\":{\"id\":\"pod-log-to-github-z2xx6-1009858533\",\"name\":\"pod-log-to-github-z2xx6.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-z2xx6\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-z2xx6\",\"startedAt\":\"2025-05-08T06:31:42Z\",\"finishedAt\":\"2025-05-08T06:31:46Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":4,\"memory\":3},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"value\":\"147\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"value\":\"0\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-z2xx6/2025/05/08/pod-log-to-github-z2xx6-system-dag-driver-1009858533/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-z2xx6-2144148312\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-z2xx6-2144148312\":{\"id\":\"pod-log-to-github-z2xx6-2144148312\",\"name\":\"pod-log-to-github-z2xx6.root\",\"displayName\":\"root\",\"type\":\"DAG\",\"templateName\":\"root\",\"templateScope\":\"local/pod-log-to-github-z2xx6\",\"phase\":\"Running\",\"boundaryID\":\"pod-log-to-github-z2xx6\",\"startedAt\":\"2025-05-08T06:31:52Z\",\"finishedAt\":null,\"progress\":\"1/2\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"147\"}]},\"children\":[\"pod-log-to-github-z2xx6-231423181\"]},\"pod-log-to-github-z2xx6-231423181\":{\"id\":\"pod-log-to-github-z2xx6-231423181\",\"name\":\"pod-log-to-github-z2xx6.root.collect-filtered-logs-driver\",\"displayName\":\"collect-filtered-logs-driver\",\"type\":\"Pod\",\"templateName\":\"system-container-driver\",\"templateScope\":\"local/pod-log-to-github-z2xx6\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-z2xx6-2144148312\",\"startedAt\":\"2025-05-08T06:31:52Z\",\"finishedAt\":\"2025-05-08T06:31:56Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":3,\"memory\":2},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\"},{\"name\":\"parent-dag-id\",\"value\":\"147\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\",\"value\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"python:3.9\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\\\",\\\"--execution_id\\\",\\\"148\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/1e12f20c-0f3e-4a06-8369-f77d6da1428f/collect-filtered-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-filtered-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"resources\\\":{}}]}\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-z2xx6/2025/05/08/pod-log-to-github-z2xx6-system-container-driver-231423181/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-z2xx6-791257680\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-z2xx6-3251966491\":{\"id\":\"pod-log-to-github-z2xx6-3251966491\",\"name\":\"pod-log-to-github-z2xx6.root.collect-filtered-logs.executor\",\"displayName\":\"executor\",\"type\":\"Pod\",\"templateName\":\"system-container-impl\",\"templateScope\":\"local/pod-log-to-github-z2xx6\",\"phase\":\"Running\",\"boundaryID\":\"pod-log-to-github-z2xx6-791257680\",\"startedAt\":\"2025-05-08T06:32:02Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"python:3.9\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\\\",\\\"--execution_id\\\",\\\"148\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/1e12f20c-0f3e-4a06-8369-f77d6da1428f/collect-filtered-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-filtered-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"resources\\\":{}}]}\"}]},\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-z2xx6-791257680\":{\"id\":\"pod-log-to-github-z2xx6-791257680\",\"name\":\"pod-log-to-github-z2xx6.root.collect-filtered-logs\",\"displayName\":\"collect-filtered-logs\",\"type\":\"DAG\",\"templateName\":\"system-container-executor\",\"templateScope\":\"local/pod-log-to-github-z2xx6\",\"phase\":\"Running\",\"boundaryID\":\"pod-log-to-github-z2xx6-2144148312\",\"startedAt\":\"2025-05-08T06:32:02Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"python:3.9\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\\\",\\\"--execution_id\\\",\\\"148\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/1e12f20c-0f3e-4a06-8369-f77d6da1428f/collect-filtered-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-filtered-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"resources\\\":{}}]}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"false\"}]},\"children\":[\"pod-log-to-github-z2xx6-3251966491\"]}},\"conditions\":[{\"type\":\"PodRunning\",\"status\":\"True\"}],\"resourcesDuration\":{\"cpu\":7,\"memory\":5},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = Unauthenticated desc = Jwt is expired: CustomError (code: 0): Error while reporting workflow resource (code: Unauthenticated, message: Jwt is expired): rpc error: code = Unauthenticated desc = Jwt is expired, {\"metadata\":{\"name\":\"pod-log-to-github-z2xx6\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"cc5e663d-85d9-4747-9066-6673e479fa1b\",\"resourceVersion\":\"6553659\",\"generation\":5,\"creationTimestamp\":\"2025-05-08T06:31:42Z\",\"labels\":{\"pipeline/runid\":\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"workflows.argoproj.io/phase\":\"Running\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-filtered-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"'  \\\\u0026\\\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str\\\\n):\\\\n\\\\n    import subprocess\\\\n    import shutil\\\\n    import os\\\\n    from datetime import datetime\\\\n\\\\n    os.environ[\\\\\\\"GIT_ASKPASS\\\\\\\"] = \\\\\\\"echo\\\\\\\"\\\\n\\\\n    report_path = report_file.path\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_path)\\\\n    if not os.path.exists(report_path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_path} does not exist\\\\\\\")\\\\n\\\\n    # HTML 파일 이름 만들기\\\\n    timestamp = datetime.now().strftime(\\\\\\\"report-%Y%m%d-%H%M%S.html\\\\\\\")\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n    report_target_path = os.path.join(repo_dir, timestamp)\\\\n\\\\n    # Git clone\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    shutil.copyfile(report_path, report_target_path)\\\\n\\\\n    # Git config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    # Git commit \\\\u0026 push\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", f\\\\\\\"Add {timestamp}\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(f\\\\\\\"Report committed and pushed to GitHub as {timestamp}.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Run of pod_log_pipeline_advance_version_at_2025-05-08T06:31:24.503Z (1347f)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:31:42Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}},{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:32:12Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-filtered-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-filtered-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-filtered-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-filtered-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-filtered-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-filtered-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:31:42Z\",\"finishedAt\":null,\"progress\":\"2/3\",\"nodes\":{\"pod-log-to-github-z2xx6\":{\"id\":\"pod-log-to-github-z2xx6\",\"name\":\"pod-log-to-github-z2xx6\",\"displayName\":\"pod-log-to-github-z2xx6\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-z2xx6\",\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:31:42Z\",\"finishedAt\":null,\"progress\":\"2/3\",\"children\":[\"pod-log-to-github-z2xx6-1009858533\"]},\"pod-log-to-github-z2xx6-1009858533\":{\"id\":\"pod-log-to-github-z2xx6-1009858533\",\"name\":\"pod-log-to-github-z2xx6.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-z2xx6\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-z2xx6\",\"startedAt\":\"2025-05-08T06:31:42Z\",\"finishedAt\":\"2025-05-08T06:31:46Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":4,\"memory\":3},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"value\":\"147\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"value\":\"0\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-z2xx6/2025/05/08/pod-log-to-github-z2xx6-system-dag-driver-1009858533/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-z2xx6-2144148312\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-z2xx6-2144148312\":{\"id\":\"pod-log-to-github-z2xx6-2144148312\",\"name\":\"pod-log-to-github-z2xx6.root\",\"displayName\":\"root\",\"type\":\"DAG\",\"templateName\":\"root\",\"templateScope\":\"local/pod-log-to-github-z2xx6\",\"phase\":\"Running\",\"boundaryID\":\"pod-log-to-github-z2xx6\",\"startedAt\":\"2025-05-08T06:31:52Z\",\"finishedAt\":null,\"progress\":\"1/2\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"147\"}]},\"children\":[\"pod-log-to-github-z2xx6-231423181\"]},\"pod-log-to-github-z2xx6-231423181\":{\"id\":\"pod-log-to-github-z2xx6-231423181\",\"name\":\"pod-log-to-github-z2xx6.root.collect-filtered-logs-driver\",\"displayName\":\"collect-filtered-logs-driver\",\"type\":\"Pod\",\"templateName\":\"system-container-driver\",\"templateScope\":\"local/pod-log-to-github-z2xx6\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-z2xx6-2144148312\",\"startedAt\":\"2025-05-08T06:31:52Z\",\"finishedAt\":\"2025-05-08T06:31:56Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":3,\"memory\":2},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\"},{\"name\":\"parent-dag-id\",\"value\":\"147\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\",\"value\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"python:3.9\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\\\",\\\"--execution_id\\\",\\\"148\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/1e12f20c-0f3e-4a06-8369-f77d6da1428f/collect-filtered-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-filtered-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"resources\\\":{}}]}\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-z2xx6/2025/05/08/pod-log-to-github-z2xx6-system-container-driver-231423181/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-z2xx6-791257680\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-z2xx6-3251966491\":{\"id\":\"pod-log-to-github-z2xx6-3251966491\",\"name\":\"pod-log-to-github-z2xx6.root.collect-filtered-logs.executor\",\"displayName\":\"executor\",\"type\":\"Pod\",\"templateName\":\"system-container-impl\",\"templateScope\":\"local/pod-log-to-github-z2xx6\",\"phase\":\"Running\",\"boundaryID\":\"pod-log-to-github-z2xx6-791257680\",\"startedAt\":\"2025-05-08T06:32:02Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"python:3.9\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\\\",\\\"--execution_id\\\",\\\"148\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/1e12f20c-0f3e-4a06-8369-f77d6da1428f/collect-filtered-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-filtered-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"resources\\\":{}}]}\"}]},\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-z2xx6-791257680\":{\"id\":\"pod-log-to-github-z2xx6-791257680\",\"name\":\"pod-log-to-github-z2xx6.root.collect-filtered-logs\",\"displayName\":\"collect-filtered-logs\",\"type\":\"DAG\",\"templateName\":\"system-container-executor\",\"templateScope\":\"local/pod-log-to-github-z2xx6\",\"phase\":\"Running\",\"boundaryID\":\"pod-log-to-github-z2xx6-2144148312\",\"startedAt\":\"2025-05-08T06:32:02Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"python:3.9\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"1e12f20c-0f3e-4a06-8369-f77d6da1428f\\\",\\\"--execution_id\\\",\\\"148\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/1e12f20c-0f3e-4a06-8369-f77d6da1428f/collect-filtered-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-filtered-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"resources\\\":{}}]}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"false\"}]},\"children\":[\"pod-log-to-github-z2xx6-3251966491\"]}},\"conditions\":[{\"type\":\"PodRunning\",\"status\":\"True\"}],\"resourcesDuration\":{\"cpu\":7,\"memory\":5},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = Unauthenticated desc = Jwt is expired"
time="2025-05-08T06:35:45Z" level=error msg="Transient failure while syncing resource (kubeflow-user-example-com/pod-log-to-github-gmf4s): CustomError (code: 0): Syncing Workflow (pod-log-to-github-gmf4s): transient failure: CustomError (code: 0): Error while reporting workflow resource (code: Unauthenticated, message: Jwt is expired): rpc error: code = Unauthenticated desc = Jwt is expired, {\"metadata\":{\"name\":\"pod-log-to-github-gmf4s\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"fae6089a-6b5b-47d7-a781-1dc96b5da279\",\"resourceVersion\":\"6554926\",\"generation\":2,\"creationTimestamp\":\"2025-05-08T06:35:44Z\",\"labels\":{\"pipeline/runid\":\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"workflows.argoproj.io/phase\":\"Running\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-filtered-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"'  \\\\u0026\\\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str\\\\n):\\\\n\\\\n    import subprocess\\\\n    import shutil\\\\n    import os\\\\n    from datetime import datetime\\\\n\\\\n    os.environ[\\\\\\\"GIT_ASKPASS\\\\\\\"] = \\\\\\\"echo\\\\\\\"\\\\n\\\\n    report_path = report_file.path\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_path)\\\\n    if not os.path.exists(report_path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_path} does not exist\\\\\\\")\\\\n\\\\n    # HTML 파일 이름 만들기\\\\n    timestamp = datetime.now().strftime(\\\\\\\"report-%Y%m%d-%H%M%S.html\\\\\\\")\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n    report_target_path = os.path.join(repo_dir, timestamp)\\\\n\\\\n    # Git clone\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    shutil.copyfile(report_path, report_target_path)\\\\n\\\\n    # Git config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    # Git commit \\\\u0026 push\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", f\\\\\\\"Add {timestamp}\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(f\\\\\\\"Report committed and pushed to GitHub as {timestamp}.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Clone of Run of pod_log_pipeline_advance_version_at_2025-05-08T06:31:24.503Z (1347f)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:35:44Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}},{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:35:44Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-filtered-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-filtered-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-filtered-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-filtered-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-filtered-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-filtered-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:35:44Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"nodes\":{\"pod-log-to-github-gmf4s\":{\"id\":\"pod-log-to-github-gmf4s\",\"name\":\"pod-log-to-github-gmf4s\",\"displayName\":\"pod-log-to-github-gmf4s\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-gmf4s\",\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:35:44Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"children\":[\"pod-log-to-github-gmf4s-2252566396\"]},\"pod-log-to-github-gmf4s-2252566396\":{\"id\":\"pod-log-to-github-gmf4s-2252566396\",\"name\":\"pod-log-to-github-gmf4s.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-gmf4s\",\"phase\":\"Pending\",\"boundaryID\":\"pod-log-to-github-gmf4s\",\"startedAt\":\"2025-05-08T06:35:44Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]}}},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = Unauthenticated desc = Jwt is expired: CustomError (code: 0): Error while reporting workflow resource (code: Unauthenticated, message: Jwt is expired): rpc error: code = Unauthenticated desc = Jwt is expired, {\"metadata\":{\"name\":\"pod-log-to-github-gmf4s\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"fae6089a-6b5b-47d7-a781-1dc96b5da279\",\"resourceVersion\":\"6554926\",\"generation\":2,\"creationTimestamp\":\"2025-05-08T06:35:44Z\",\"labels\":{\"pipeline/runid\":\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"workflows.argoproj.io/phase\":\"Running\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-filtered-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"'  \\\\u0026\\\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str\\\\n):\\\\n\\\\n    import subprocess\\\\n    import shutil\\\\n    import os\\\\n    from datetime import datetime\\\\n\\\\n    os.environ[\\\\\\\"GIT_ASKPASS\\\\\\\"] = \\\\\\\"echo\\\\\\\"\\\\n\\\\n    report_path = report_file.path\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_path)\\\\n    if not os.path.exists(report_path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_path} does not exist\\\\\\\")\\\\n\\\\n    # HTML 파일 이름 만들기\\\\n    timestamp = datetime.now().strftime(\\\\\\\"report-%Y%m%d-%H%M%S.html\\\\\\\")\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n    report_target_path = os.path.join(repo_dir, timestamp)\\\\n\\\\n    # Git clone\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    shutil.copyfile(report_path, report_target_path)\\\\n\\\\n    # Git config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    # Git commit \\\\u0026 push\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", f\\\\\\\"Add {timestamp}\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(f\\\\\\\"Report committed and pushed to GitHub as {timestamp}.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Clone of Run of pod_log_pipeline_advance_version_at_2025-05-08T06:31:24.503Z (1347f)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:35:44Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}},{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:35:44Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-filtered-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-filtered-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-filtered-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-filtered-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-filtered-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-filtered-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:35:44Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"nodes\":{\"pod-log-to-github-gmf4s\":{\"id\":\"pod-log-to-github-gmf4s\",\"name\":\"pod-log-to-github-gmf4s\",\"displayName\":\"pod-log-to-github-gmf4s\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-gmf4s\",\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:35:44Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"children\":[\"pod-log-to-github-gmf4s-2252566396\"]},\"pod-log-to-github-gmf4s-2252566396\":{\"id\":\"pod-log-to-github-gmf4s-2252566396\",\"name\":\"pod-log-to-github-gmf4s.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-gmf4s\",\"phase\":\"Pending\",\"boundaryID\":\"pod-log-to-github-gmf4s\",\"startedAt\":\"2025-05-08T06:35:44Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]}}},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = Unauthenticated desc = Jwt is expired"
time="2025-05-08T06:35:49Z" level=error msg="Transient failure while syncing resource (kubeflow-user-example-com/pod-log-to-github-gmf4s): CustomError (code: 0): Syncing Workflow (pod-log-to-github-gmf4s): transient failure: CustomError (code: 0): Error while reporting workflow resource (code: Unauthenticated, message: Jwt is expired): rpc error: code = Unauthenticated desc = Jwt is expired, {\"metadata\":{\"name\":\"pod-log-to-github-gmf4s\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"fae6089a-6b5b-47d7-a781-1dc96b5da279\",\"resourceVersion\":\"6554926\",\"generation\":2,\"creationTimestamp\":\"2025-05-08T06:35:44Z\",\"labels\":{\"pipeline/runid\":\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"workflows.argoproj.io/phase\":\"Running\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-filtered-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"'  \\\\u0026\\\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str\\\\n):\\\\n\\\\n    import subprocess\\\\n    import shutil\\\\n    import os\\\\n    from datetime import datetime\\\\n\\\\n    os.environ[\\\\\\\"GIT_ASKPASS\\\\\\\"] = \\\\\\\"echo\\\\\\\"\\\\n\\\\n    report_path = report_file.path\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_path)\\\\n    if not os.path.exists(report_path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_path} does not exist\\\\\\\")\\\\n\\\\n    # HTML 파일 이름 만들기\\\\n    timestamp = datetime.now().strftime(\\\\\\\"report-%Y%m%d-%H%M%S.html\\\\\\\")\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n    report_target_path = os.path.join(repo_dir, timestamp)\\\\n\\\\n    # Git clone\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    shutil.copyfile(report_path, report_target_path)\\\\n\\\\n    # Git config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    # Git commit \\\\u0026 push\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", f\\\\\\\"Add {timestamp}\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(f\\\\\\\"Report committed and pushed to GitHub as {timestamp}.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Clone of Run of pod_log_pipeline_advance_version_at_2025-05-08T06:31:24.503Z (1347f)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:35:44Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}},{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:35:44Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-filtered-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-filtered-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-filtered-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-filtered-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-filtered-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-filtered-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:35:44Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"nodes\":{\"pod-log-to-github-gmf4s\":{\"id\":\"pod-log-to-github-gmf4s\",\"name\":\"pod-log-to-github-gmf4s\",\"displayName\":\"pod-log-to-github-gmf4s\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-gmf4s\",\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:35:44Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"children\":[\"pod-log-to-github-gmf4s-2252566396\"]},\"pod-log-to-github-gmf4s-2252566396\":{\"id\":\"pod-log-to-github-gmf4s-2252566396\",\"name\":\"pod-log-to-github-gmf4s.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-gmf4s\",\"phase\":\"Pending\",\"boundaryID\":\"pod-log-to-github-gmf4s\",\"startedAt\":\"2025-05-08T06:35:44Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]}}},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = Unauthenticated desc = Jwt is expired: CustomError (code: 0): Error while reporting workflow resource (code: Unauthenticated, message: Jwt is expired): rpc error: code = Unauthenticated desc = Jwt is expired, {\"metadata\":{\"name\":\"pod-log-to-github-gmf4s\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"fae6089a-6b5b-47d7-a781-1dc96b5da279\",\"resourceVersion\":\"6554926\",\"generation\":2,\"creationTimestamp\":\"2025-05-08T06:35:44Z\",\"labels\":{\"pipeline/runid\":\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"workflows.argoproj.io/phase\":\"Running\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-filtered-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"'  \\\\u0026\\\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str\\\\n):\\\\n\\\\n    import subprocess\\\\n    import shutil\\\\n    import os\\\\n    from datetime import datetime\\\\n\\\\n    os.environ[\\\\\\\"GIT_ASKPASS\\\\\\\"] = \\\\\\\"echo\\\\\\\"\\\\n\\\\n    report_path = report_file.path\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_path)\\\\n    if not os.path.exists(report_path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_path} does not exist\\\\\\\")\\\\n\\\\n    # HTML 파일 이름 만들기\\\\n    timestamp = datetime.now().strftime(\\\\\\\"report-%Y%m%d-%H%M%S.html\\\\\\\")\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n    report_target_path = os.path.join(repo_dir, timestamp)\\\\n\\\\n    # Git clone\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    shutil.copyfile(report_path, report_target_path)\\\\n\\\\n    # Git config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    # Git commit \\\\u0026 push\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", f\\\\\\\"Add {timestamp}\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(f\\\\\\\"Report committed and pushed to GitHub as {timestamp}.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Clone of Run of pod_log_pipeline_advance_version_at_2025-05-08T06:31:24.503Z (1347f)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:35:44Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}},{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:35:44Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-filtered-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-filtered-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-filtered-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-filtered-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-filtered-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-filtered-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:35:44Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"nodes\":{\"pod-log-to-github-gmf4s\":{\"id\":\"pod-log-to-github-gmf4s\",\"name\":\"pod-log-to-github-gmf4s\",\"displayName\":\"pod-log-to-github-gmf4s\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-gmf4s\",\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:35:44Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"children\":[\"pod-log-to-github-gmf4s-2252566396\"]},\"pod-log-to-github-gmf4s-2252566396\":{\"id\":\"pod-log-to-github-gmf4s-2252566396\",\"name\":\"pod-log-to-github-gmf4s.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-gmf4s\",\"phase\":\"Pending\",\"boundaryID\":\"pod-log-to-github-gmf4s\",\"startedAt\":\"2025-05-08T06:35:44Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]}}},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = Unauthenticated desc = Jwt is expired"
time="2025-05-08T06:35:57Z" level=error msg="Transient failure while syncing resource (kubeflow-user-example-com/pod-log-to-github-gmf4s): CustomError (code: 0): Syncing Workflow (pod-log-to-github-gmf4s): transient failure: CustomError (code: 0): Error while reporting workflow resource (code: Unauthenticated, message: Jwt is expired): rpc error: code = Unauthenticated desc = Jwt is expired, {\"metadata\":{\"name\":\"pod-log-to-github-gmf4s\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"fae6089a-6b5b-47d7-a781-1dc96b5da279\",\"resourceVersion\":\"6555002\",\"generation\":3,\"creationTimestamp\":\"2025-05-08T06:35:44Z\",\"labels\":{\"pipeline/runid\":\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"workflows.argoproj.io/phase\":\"Running\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-filtered-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"'  \\\\u0026\\\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str\\\\n):\\\\n\\\\n    import subprocess\\\\n    import shutil\\\\n    import os\\\\n    from datetime import datetime\\\\n\\\\n    os.environ[\\\\\\\"GIT_ASKPASS\\\\\\\"] = \\\\\\\"echo\\\\\\\"\\\\n\\\\n    report_path = report_file.path\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_path)\\\\n    if not os.path.exists(report_path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_path} does not exist\\\\\\\")\\\\n\\\\n    # HTML 파일 이름 만들기\\\\n    timestamp = datetime.now().strftime(\\\\\\\"report-%Y%m%d-%H%M%S.html\\\\\\\")\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n    report_target_path = os.path.join(repo_dir, timestamp)\\\\n\\\\n    # Git clone\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    shutil.copyfile(report_path, report_target_path)\\\\n\\\\n    # Git config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    # Git commit \\\\u0026 push\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", f\\\\\\\"Add {timestamp}\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(f\\\\\\\"Report committed and pushed to GitHub as {timestamp}.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Clone of Run of pod_log_pipeline_advance_version_at_2025-05-08T06:31:24.503Z (1347f)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:35:44Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}},{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:35:54Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-filtered-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-filtered-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-filtered-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-filtered-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-filtered-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-filtered-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:35:44Z\",\"finishedAt\":null,\"progress\":\"1/2\",\"nodes\":{\"pod-log-to-github-gmf4s\":{\"id\":\"pod-log-to-github-gmf4s\",\"name\":\"pod-log-to-github-gmf4s\",\"displayName\":\"pod-log-to-github-gmf4s\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-gmf4s\",\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:35:44Z\",\"finishedAt\":null,\"progress\":\"1/2\",\"children\":[\"pod-log-to-github-gmf4s-2252566396\"]},\"pod-log-to-github-gmf4s-2119004180\":{\"id\":\"pod-log-to-github-gmf4s-2119004180\",\"name\":\"pod-log-to-github-gmf4s.root.collect-filtered-logs-driver\",\"displayName\":\"collect-filtered-logs-driver\",\"type\":\"Pod\",\"templateName\":\"system-container-driver\",\"templateScope\":\"local/pod-log-to-github-gmf4s\",\"phase\":\"Pending\",\"boundaryID\":\"pod-log-to-github-gmf4s-4161926427\",\"startedAt\":\"2025-05-08T06:35:54Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\"},{\"name\":\"parent-dag-id\",\"value\":\"150\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\",\"value\":\"\"}]}},\"pod-log-to-github-gmf4s-2252566396\":{\"id\":\"pod-log-to-github-gmf4s-2252566396\",\"name\":\"pod-log-to-github-gmf4s.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-gmf4s\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-gmf4s\",\"startedAt\":\"2025-05-08T06:35:44Z\",\"finishedAt\":\"2025-05-08T06:35:48Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":4,\"memory\":3},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"value\":\"150\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"value\":\"0\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-gmf4s/2025/05/08/pod-log-to-github-gmf4s-system-dag-driver-2252566396/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-gmf4s-4161926427\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-gmf4s-4161926427\":{\"id\":\"pod-log-to-github-gmf4s-4161926427\",\"name\":\"pod-log-to-github-gmf4s.root\",\"displayName\":\"root\",\"type\":\"DAG\",\"templateName\":\"root\",\"templateScope\":\"local/pod-log-to-github-gmf4s\",\"phase\":\"Running\",\"boundaryID\":\"pod-log-to-github-gmf4s\",\"startedAt\":\"2025-05-08T06:35:54Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"150\"}]},\"children\":[\"pod-log-to-github-gmf4s-2119004180\"]}},\"conditions\":[{\"type\":\"PodRunning\",\"status\":\"False\"}],\"resourcesDuration\":{\"cpu\":4,\"memory\":3},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = Unauthenticated desc = Jwt is expired: CustomError (code: 0): Error while reporting workflow resource (code: Unauthenticated, message: Jwt is expired): rpc error: code = Unauthenticated desc = Jwt is expired, {\"metadata\":{\"name\":\"pod-log-to-github-gmf4s\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"fae6089a-6b5b-47d7-a781-1dc96b5da279\",\"resourceVersion\":\"6555002\",\"generation\":3,\"creationTimestamp\":\"2025-05-08T06:35:44Z\",\"labels\":{\"pipeline/runid\":\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"workflows.argoproj.io/phase\":\"Running\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-filtered-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"'  \\\\u0026\\\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str\\\\n):\\\\n\\\\n    import subprocess\\\\n    import shutil\\\\n    import os\\\\n    from datetime import datetime\\\\n\\\\n    os.environ[\\\\\\\"GIT_ASKPASS\\\\\\\"] = \\\\\\\"echo\\\\\\\"\\\\n\\\\n    report_path = report_file.path\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_path)\\\\n    if not os.path.exists(report_path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_path} does not exist\\\\\\\")\\\\n\\\\n    # HTML 파일 이름 만들기\\\\n    timestamp = datetime.now().strftime(\\\\\\\"report-%Y%m%d-%H%M%S.html\\\\\\\")\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n    report_target_path = os.path.join(repo_dir, timestamp)\\\\n\\\\n    # Git clone\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    shutil.copyfile(report_path, report_target_path)\\\\n\\\\n    # Git config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    # Git commit \\\\u0026 push\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", f\\\\\\\"Add {timestamp}\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(f\\\\\\\"Report committed and pushed to GitHub as {timestamp}.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Clone of Run of pod_log_pipeline_advance_version_at_2025-05-08T06:31:24.503Z (1347f)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:35:44Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}},{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:35:54Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-filtered-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-filtered-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-filtered-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-filtered-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-filtered-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-filtered-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:35:44Z\",\"finishedAt\":null,\"progress\":\"1/2\",\"nodes\":{\"pod-log-to-github-gmf4s\":{\"id\":\"pod-log-to-github-gmf4s\",\"name\":\"pod-log-to-github-gmf4s\",\"displayName\":\"pod-log-to-github-gmf4s\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-gmf4s\",\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:35:44Z\",\"finishedAt\":null,\"progress\":\"1/2\",\"children\":[\"pod-log-to-github-gmf4s-2252566396\"]},\"pod-log-to-github-gmf4s-2119004180\":{\"id\":\"pod-log-to-github-gmf4s-2119004180\",\"name\":\"pod-log-to-github-gmf4s.root.collect-filtered-logs-driver\",\"displayName\":\"collect-filtered-logs-driver\",\"type\":\"Pod\",\"templateName\":\"system-container-driver\",\"templateScope\":\"local/pod-log-to-github-gmf4s\",\"phase\":\"Pending\",\"boundaryID\":\"pod-log-to-github-gmf4s-4161926427\",\"startedAt\":\"2025-05-08T06:35:54Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\"},{\"name\":\"parent-dag-id\",\"value\":\"150\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\",\"value\":\"\"}]}},\"pod-log-to-github-gmf4s-2252566396\":{\"id\":\"pod-log-to-github-gmf4s-2252566396\",\"name\":\"pod-log-to-github-gmf4s.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-gmf4s\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-gmf4s\",\"startedAt\":\"2025-05-08T06:35:44Z\",\"finishedAt\":\"2025-05-08T06:35:48Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":4,\"memory\":3},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"value\":\"150\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"value\":\"0\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-gmf4s/2025/05/08/pod-log-to-github-gmf4s-system-dag-driver-2252566396/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-gmf4s-4161926427\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-gmf4s-4161926427\":{\"id\":\"pod-log-to-github-gmf4s-4161926427\",\"name\":\"pod-log-to-github-gmf4s.root\",\"displayName\":\"root\",\"type\":\"DAG\",\"templateName\":\"root\",\"templateScope\":\"local/pod-log-to-github-gmf4s\",\"phase\":\"Running\",\"boundaryID\":\"pod-log-to-github-gmf4s\",\"startedAt\":\"2025-05-08T06:35:54Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"150\"}]},\"children\":[\"pod-log-to-github-gmf4s-2119004180\"]}},\"conditions\":[{\"type\":\"PodRunning\",\"status\":\"False\"}],\"resourcesDuration\":{\"cpu\":4,\"memory\":3},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = Unauthenticated desc = Jwt is expired"
time="2025-05-08T06:37:01Z" level=error msg="Transient failure while syncing resource (kubeflow-user-example-com/pod-log-to-github-gmf4s): CustomError (code: 0): Syncing Workflow (pod-log-to-github-gmf4s): transient failure: CustomError (code: 0): Error while reporting workflow resource (code: Unauthenticated, message: Jwt is expired): rpc error: code = Unauthenticated desc = Jwt is expired, {\"metadata\":{\"name\":\"pod-log-to-github-gmf4s\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"fae6089a-6b5b-47d7-a781-1dc96b5da279\",\"resourceVersion\":\"6555159\",\"generation\":5,\"creationTimestamp\":\"2025-05-08T06:35:44Z\",\"labels\":{\"pipeline/runid\":\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"workflows.argoproj.io/phase\":\"Running\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-filtered-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"'  \\\\u0026\\\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str\\\\n):\\\\n\\\\n    import subprocess\\\\n    import shutil\\\\n    import os\\\\n    from datetime import datetime\\\\n\\\\n    os.environ[\\\\\\\"GIT_ASKPASS\\\\\\\"] = \\\\\\\"echo\\\\\\\"\\\\n\\\\n    report_path = report_file.path\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_path)\\\\n    if not os.path.exists(report_path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_path} does not exist\\\\\\\")\\\\n\\\\n    # HTML 파일 이름 만들기\\\\n    timestamp = datetime.now().strftime(\\\\\\\"report-%Y%m%d-%H%M%S.html\\\\\\\")\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n    report_target_path = os.path.join(repo_dir, timestamp)\\\\n\\\\n    # Git clone\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    shutil.copyfile(report_path, report_target_path)\\\\n\\\\n    # Git config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    # Git commit \\\\u0026 push\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", f\\\\\\\"Add {timestamp}\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(f\\\\\\\"Report committed and pushed to GitHub as {timestamp}.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Clone of Run of pod_log_pipeline_advance_version_at_2025-05-08T06:31:24.503Z (1347f)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:35:44Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}},{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:36:14Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-filtered-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-filtered-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-filtered-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-filtered-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-filtered-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-filtered-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:35:44Z\",\"finishedAt\":null,\"progress\":\"2/3\",\"nodes\":{\"pod-log-to-github-gmf4s\":{\"id\":\"pod-log-to-github-gmf4s\",\"name\":\"pod-log-to-github-gmf4s\",\"displayName\":\"pod-log-to-github-gmf4s\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-gmf4s\",\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:35:44Z\",\"finishedAt\":null,\"progress\":\"2/3\",\"children\":[\"pod-log-to-github-gmf4s-2252566396\"]},\"pod-log-to-github-gmf4s-1948391354\":{\"id\":\"pod-log-to-github-gmf4s-1948391354\",\"name\":\"pod-log-to-github-gmf4s.root.collect-filtered-logs.executor\",\"displayName\":\"executor\",\"type\":\"Pod\",\"templateName\":\"system-container-impl\",\"templateScope\":\"local/pod-log-to-github-gmf4s\",\"phase\":\"Running\",\"boundaryID\":\"pod-log-to-github-gmf4s-3170595587\",\"startedAt\":\"2025-05-08T06:36:04Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"python:3.9\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"b9842c1b-438d-410c-ad18-f01e0145433b\\\",\\\"--execution_id\\\",\\\"151\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b9842c1b-438d-410c-ad18-f01e0145433b/collect-filtered-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-filtered-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"resources\\\":{}}]}\"}]},\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-gmf4s-2119004180\":{\"id\":\"pod-log-to-github-gmf4s-2119004180\",\"name\":\"pod-log-to-github-gmf4s.root.collect-filtered-logs-driver\",\"displayName\":\"collect-filtered-logs-driver\",\"type\":\"Pod\",\"templateName\":\"system-container-driver\",\"templateScope\":\"local/pod-log-to-github-gmf4s\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-gmf4s-4161926427\",\"startedAt\":\"2025-05-08T06:35:54Z\",\"finishedAt\":\"2025-05-08T06:35:58Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":3,\"memory\":2},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\"},{\"name\":\"parent-dag-id\",\"value\":\"150\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\",\"value\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"python:3.9\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"b9842c1b-438d-410c-ad18-f01e0145433b\\\",\\\"--execution_id\\\",\\\"151\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b9842c1b-438d-410c-ad18-f01e0145433b/collect-filtered-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-filtered-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"resources\\\":{}}]}\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-gmf4s/2025/05/08/pod-log-to-github-gmf4s-system-container-driver-2119004180/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-gmf4s-3170595587\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-gmf4s-2252566396\":{\"id\":\"pod-log-to-github-gmf4s-2252566396\",\"name\":\"pod-log-to-github-gmf4s.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-gmf4s\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-gmf4s\",\"startedAt\":\"2025-05-08T06:35:44Z\",\"finishedAt\":\"2025-05-08T06:35:48Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":4,\"memory\":3},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"value\":\"150\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"value\":\"0\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-gmf4s/2025/05/08/pod-log-to-github-gmf4s-system-dag-driver-2252566396/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-gmf4s-4161926427\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-gmf4s-3170595587\":{\"id\":\"pod-log-to-github-gmf4s-3170595587\",\"name\":\"pod-log-to-github-gmf4s.root.collect-filtered-logs\",\"displayName\":\"collect-filtered-logs\",\"type\":\"DAG\",\"templateName\":\"system-container-executor\",\"templateScope\":\"local/pod-log-to-github-gmf4s\",\"phase\":\"Running\",\"boundaryID\":\"pod-log-to-github-gmf4s-4161926427\",\"startedAt\":\"2025-05-08T06:36:04Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"python:3.9\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"b9842c1b-438d-410c-ad18-f01e0145433b\\\",\\\"--execution_id\\\",\\\"151\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b9842c1b-438d-410c-ad18-f01e0145433b/collect-filtered-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-filtered-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"resources\\\":{}}]}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"false\"}]},\"children\":[\"pod-log-to-github-gmf4s-1948391354\"]},\"pod-log-to-github-gmf4s-4161926427\":{\"id\":\"pod-log-to-github-gmf4s-4161926427\",\"name\":\"pod-log-to-github-gmf4s.root\",\"displayName\":\"root\",\"type\":\"DAG\",\"templateName\":\"root\",\"templateScope\":\"local/pod-log-to-github-gmf4s\",\"phase\":\"Running\",\"boundaryID\":\"pod-log-to-github-gmf4s\",\"startedAt\":\"2025-05-08T06:35:54Z\",\"finishedAt\":null,\"progress\":\"1/2\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"150\"}]},\"children\":[\"pod-log-to-github-gmf4s-2119004180\"]}},\"conditions\":[{\"type\":\"PodRunning\",\"status\":\"True\"}],\"resourcesDuration\":{\"cpu\":7,\"memory\":5},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = Unauthenticated desc = Jwt is expired: CustomError (code: 0): Error while reporting workflow resource (code: Unauthenticated, message: Jwt is expired): rpc error: code = Unauthenticated desc = Jwt is expired, {\"metadata\":{\"name\":\"pod-log-to-github-gmf4s\",\"generateName\":\"pod-log-to-github-\",\"namespace\":\"kubeflow-user-example-com\",\"uid\":\"fae6089a-6b5b-47d7-a781-1dc96b5da279\",\"resourceVersion\":\"6555159\",\"generation\":5,\"creationTimestamp\":\"2025-05-08T06:35:44Z\",\"labels\":{\"pipeline/runid\":\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"workflows.argoproj.io/phase\":\"Running\"},\"annotations\":{\"pipelines.kubeflow.org/components-comp-collect-filtered-logs\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\",\"pipelines.kubeflow.org/components-comp-upload-to-github\":\"{\\\"executorLabel\\\":\\\"exec-upload-to-github\\\",\\\"inputDefinitions\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/components-root\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\",\"pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/implementations-comp-upload-to-github\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"upload_to_github\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"'  \\\\u0026\\\\u0026  python3 -m pip install --quiet --no-warn-script-location 'gitpython' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef upload_to_github(\\\\n    report_file: Input[Dataset],\\\\n    repo_url: str,\\\\n    github_token: str,\\\\n    branch: str\\\\n):\\\\n\\\\n    import subprocess\\\\n    import shutil\\\\n    import os\\\\n    from datetime import datetime\\\\n\\\\n    os.environ[\\\\\\\"GIT_ASKPASS\\\\\\\"] = \\\\\\\"echo\\\\\\\"\\\\n\\\\n    report_path = report_file.path\\\\n    print(\\\\\\\"Checking if report exists at:\\\\\\\", report_path)\\\\n    if not os.path.exists(report_path):\\\\n        raise FileNotFoundError(f\\\\\\\"{report_path} does not exist\\\\\\\")\\\\n\\\\n    # HTML 파일 이름 만들기\\\\n    timestamp = datetime.now().strftime(\\\\\\\"report-%Y%m%d-%H%M%S.html\\\\\\\")\\\\n    safe_repo_url = repo_url.replace(\\\\\\\"https://\\\\\\\", f\\\\\\\"https://{github_token}@\\\\\\\")\\\\n    repo_dir = \\\\\\\"/tmp/repo\\\\\\\"\\\\n    report_target_path = os.path.join(repo_dir, timestamp)\\\\n\\\\n    # Git clone\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"clone\\\\\\\", \\\\\\\"--branch\\\\\\\", branch, safe_repo_url, repo_dir], check=True)\\\\n    shutil.copyfile(report_path, report_target_path)\\\\n\\\\n    # Git config\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.email\\\\\\\", \\\\\\\"ci@example.com\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"config\\\\\\\", \\\\\\\"user.name\\\\\\\", \\\\\\\"CI Bot\\\\\\\"], check=True)\\\\n\\\\n    # Git commit \\\\u0026 push\\\\n    subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"add\\\\\\\", \\\\\\\".\\\\\\\"], check=True)\\\\n    result = subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"status\\\\\\\", \\\\\\\"--porcelain\\\\\\\"], capture_output=True, text=True)\\\\n\\\\n    if result.stdout.strip():\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"commit\\\\\\\", \\\\\\\"-m\\\\\\\", f\\\\\\\"Add {timestamp}\\\\\\\"], check=True)\\\\n        subprocess.run([\\\\\\\"git\\\\\\\", \\\\\\\"-C\\\\\\\", repo_dir, \\\\\\\"push\\\\\\\"], check=True)\\\\n        print(f\\\\\\\"Report committed and pushed to GitHub as {timestamp}.\\\\\\\")\\\\n    else:\\\\n        print(\\\\\\\"No changes to commit. Skipping.\\\\\\\")\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\",\"pipelines.kubeflow.org/run_name\":\"Clone of Run of pod_log_pipeline_advance_version_at_2025-05-08T06:31:24.503Z (1347f)\",\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"managedFields\":[{\"manager\":\"apiserver\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:35:44Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pipelines.kubeflow.org/components-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/components-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/components-root\":{},\"f:pipelines.kubeflow.org/implementations-comp-collect-filtered-logs\":{},\"f:pipelines.kubeflow.org/implementations-comp-upload-to-github\":{},\"f:pipelines.kubeflow.org/run_name\":{}},\"f:generateName\":{},\"f:labels\":{\".\":{},\"f:pipeline/runid\":{}}},\"f:spec\":{}}},{\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"apiVersion\":\"argoproj.io/v1alpha1\",\"time\":\"2025-05-08T06:36:14Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:workflows.argoproj.io/pod-name-format\":{}},\"f:labels\":{\"f:workflows.argoproj.io/phase\":{}}},\"f:status\":{}}}]},\"spec\":{\"templates\":[{\"name\":\"system-container-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"task\"},{\"name\":\"container\"},{\"name\":\"parent-dag-id\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"CONTAINER\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--container\",\"{{inputs.parameters.container}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--cached_decision_path\",\"{{outputs.parameters.cached-decision.path}}\",\"--pod_spec_patch_path\",\"{{outputs.parameters.pod-spec-patch.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\",\"--kubernetes_config\",\"{{inputs.parameters.kubernetes-config}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"system-container-executor\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"},{\"name\":\"cached-decision\",\"default\":\"false\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"executor\",\"template\":\"system-container-impl\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{inputs.parameters.pod-spec-patch}}\"}]},\"when\":\"{{inputs.parameters.cached-decision}} != true\"}]}},{\"name\":\"system-container-impl\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/should-be-overridden-during-runtime\",\"command\":[\"should-be-overridden-during-runtime\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"metadata-grpc-configmap\",\"optional\":true}}],\"env\":[{\"name\":\"KFP_POD_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.name\"}}},{\"name\":\"KFP_POD_UID\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.uid\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]},\"volumes\":[{\"name\":\"kfp-launcher\",\"emptyDir\":{}}],\"initContainers\":[{\"name\":\"kfp-launcher\",\"image\":\"gcr.io/ml-pipeline/kfp-launcher@sha256:8fe5e6e4718f20b021736022ad3741ddf2abd82aa58c86ae13e89736fdc3f08f\",\"command\":[\"launcher-v2\",\"--copy\",\"/kfp-launcher/launch\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"},\"requests\":{\"cpu\":\"100m\"}},\"volumeMounts\":[{\"name\":\"kfp-launcher\",\"mountPath\":\"/kfp-launcher\"}]}],\"podSpecPatch\":\"{{inputs.parameters.pod-spec-patch}}\"},{\"name\":\"root\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\"}]},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"collect-filtered-logs-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-collect-filtered-logs}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-collect-filtered-logs}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]}},{\"name\":\"collect-filtered-logs\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.collect-filtered-logs-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"collect-filtered-logs-driver.Succeeded\"},{\"name\":\"upload-to-github-driver\",\"template\":\"system-container-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-comp-upload-to-github}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}\"},{\"name\":\"container\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/implementations-comp-upload-to-github}}\"},{\"name\":\"parent-dag-id\",\"value\":\"{{inputs.parameters.parent-dag-id}}\"}]},\"depends\":\"collect-filtered-logs.Succeeded\"},{\"name\":\"upload-to-github\",\"template\":\"system-container-executor\",\"arguments\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.pod-spec-patch}}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"{{tasks.upload-to-github-driver.outputs.parameters.cached-decision}}\"}]},\"depends\":\"upload-to-github-driver.Succeeded\"}]}},{\"name\":\"system-dag-driver\",\"inputs\":{\"parameters\":[{\"name\":\"component\"},{\"name\":\"runtime-config\",\"default\":\"\"},{\"name\":\"task\",\"default\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}]},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"\",\"image\":\"gcr.io/ml-pipeline/kfp-driver@sha256:3c0665cd36aa87e4359a4c8b6271dcba5bdd817815cd0496ed12eb5dde5fd2ec\",\"command\":[\"driver\"],\"args\":[\"--type\",\"{{inputs.parameters.driver-type}}\",\"--pipeline_name\",\"pod-log-to-github\",\"--run_id\",\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"--dag_execution_id\",\"{{inputs.parameters.parent-dag-id}}\",\"--component\",\"{{inputs.parameters.component}}\",\"--task\",\"{{inputs.parameters.task}}\",\"--runtime_config\",\"{{inputs.parameters.runtime-config}}\",\"--iteration_index\",\"{{inputs.parameters.iteration-index}}\",\"--execution_id_path\",\"{{outputs.parameters.execution-id.path}}\",\"--iteration_count_path\",\"{{outputs.parameters.iteration-count.path}}\",\"--condition_path\",\"{{outputs.parameters.condition.path}}\"],\"resources\":{\"limits\":{\"cpu\":\"500m\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"}}}},{\"name\":\"entrypoint\",\"inputs\":{},\"outputs\":{},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"dag\":{\"tasks\":[{\"name\":\"root-driver\",\"template\":\"system-dag-driver\",\"arguments\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{{workflow.annotations.pipelines.kubeflow.org/components-root}}\"},{\"name\":\"runtime-config\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"driver-type\",\"value\":\"ROOT_DAG\"}]}},{\"name\":\"root\",\"template\":\"root\",\"arguments\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"{{tasks.root-driver.outputs.parameters.execution-id}}\"},{\"name\":\"condition\",\"value\":\"\"}]},\"depends\":\"root-driver.Succeeded\"}]}}],\"entrypoint\":\"entrypoint\",\"arguments\":{},\"serviceAccountName\":\"default-editor\",\"podMetadata\":{\"annotations\":{\"pipelines.kubeflow.org/v2_component\":\"true\"},\"labels\":{\"pipeline/runid\":\"b9842c1b-438d-410c-ad18-f01e0145433b\",\"pipelines.kubeflow.org/v2_component\":\"true\"}}},\"status\":{\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:35:44Z\",\"finishedAt\":null,\"progress\":\"2/3\",\"nodes\":{\"pod-log-to-github-gmf4s\":{\"id\":\"pod-log-to-github-gmf4s\",\"name\":\"pod-log-to-github-gmf4s\",\"displayName\":\"pod-log-to-github-gmf4s\",\"type\":\"DAG\",\"templateName\":\"entrypoint\",\"templateScope\":\"local/pod-log-to-github-gmf4s\",\"phase\":\"Running\",\"startedAt\":\"2025-05-08T06:35:44Z\",\"finishedAt\":null,\"progress\":\"2/3\",\"children\":[\"pod-log-to-github-gmf4s-2252566396\"]},\"pod-log-to-github-gmf4s-1948391354\":{\"id\":\"pod-log-to-github-gmf4s-1948391354\",\"name\":\"pod-log-to-github-gmf4s.root.collect-filtered-logs.executor\",\"displayName\":\"executor\",\"type\":\"Pod\",\"templateName\":\"system-container-impl\",\"templateScope\":\"local/pod-log-to-github-gmf4s\",\"phase\":\"Running\",\"boundaryID\":\"pod-log-to-github-gmf4s-3170595587\",\"startedAt\":\"2025-05-08T06:36:04Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"python:3.9\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"b9842c1b-438d-410c-ad18-f01e0145433b\\\",\\\"--execution_id\\\",\\\"151\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b9842c1b-438d-410c-ad18-f01e0145433b/collect-filtered-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-filtered-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"resources\\\":{}}]}\"}]},\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-gmf4s-2119004180\":{\"id\":\"pod-log-to-github-gmf4s-2119004180\",\"name\":\"pod-log-to-github-gmf4s.root.collect-filtered-logs-driver\",\"displayName\":\"collect-filtered-logs-driver\",\"type\":\"Pod\",\"templateName\":\"system-container-driver\",\"templateScope\":\"local/pod-log-to-github-gmf4s\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-gmf4s-4161926427\",\"startedAt\":\"2025-05-08T06:35:54Z\",\"finishedAt\":\"2025-05-08T06:35:58Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":3,\"memory\":2},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"executorLabel\\\":\\\"exec-collect-filtered-logs\\\",\\\"outputDefinitions\\\":{\\\"artifacts\\\":{\\\"output_report\\\":{\\\"artifactType\\\":{\\\"schemaTitle\\\":\\\"system.Dataset\\\",\\\"schemaVersion\\\":\\\"0.0.1\\\"}}}}}\"},{\"name\":\"task\",\"value\":\"{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}}\"},{\"name\":\"container\",\"value\":\"{\\\"args\\\":[\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"command\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\"],\\\"image\\\":\\\"python:3.9\\\"}\"},{\"name\":\"parent-dag-id\",\"value\":\"150\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"kubernetes-config\",\"default\":\"\",\"value\":\"\"}]},\"outputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"python:3.9\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"b9842c1b-438d-410c-ad18-f01e0145433b\\\",\\\"--execution_id\\\",\\\"151\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b9842c1b-438d-410c-ad18-f01e0145433b/collect-filtered-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-filtered-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"resources\\\":{}}]}\",\"valueFrom\":{\"path\":\"/tmp/outputs/pod-spec-patch\",\"default\":\"\"}},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"false\",\"valueFrom\":{\"path\":\"/tmp/outputs/cached-decision\",\"default\":\"false\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-gmf4s/2025/05/08/pod-log-to-github-gmf4s-system-container-driver-2119004180/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-gmf4s-3170595587\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-gmf4s-2252566396\":{\"id\":\"pod-log-to-github-gmf4s-2252566396\",\"name\":\"pod-log-to-github-gmf4s.root-driver\",\"displayName\":\"root-driver\",\"type\":\"Pod\",\"templateName\":\"system-dag-driver\",\"templateScope\":\"local/pod-log-to-github-gmf4s\",\"phase\":\"Succeeded\",\"boundaryID\":\"pod-log-to-github-gmf4s\",\"startedAt\":\"2025-05-08T06:35:44Z\",\"finishedAt\":\"2025-05-08T06:35:48Z\",\"progress\":\"1/1\",\"resourcesDuration\":{\"cpu\":4,\"memory\":3},\"inputs\":{\"parameters\":[{\"name\":\"component\",\"value\":\"{\\\"dag\\\":{\\\"tasks\\\":{\\\"collect-filtered-logs\\\":{\\\"cachingOptions\\\":{},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-collect-filtered-logs\\\"},\\\"taskInfo\\\":{\\\"name\\\":\\\"collect-filtered-logs\\\"}},\\\"upload-to-github\\\":{\\\"cachingOptions\\\":{\\\"enableCache\\\":true},\\\"componentRef\\\":{\\\"name\\\":\\\"comp-upload-to-github\\\"},\\\"dependentTasks\\\":[\\\"collect-filtered-logs\\\"],\\\"inputs\\\":{\\\"artifacts\\\":{\\\"report_file\\\":{\\\"taskOutputArtifact\\\":{\\\"outputArtifactKey\\\":\\\"output_report\\\",\\\"producerTask\\\":\\\"collect-filtered-logs\\\"}}},\\\"parameters\\\":{\\\"branch\\\":{\\\"componentInputParameter\\\":\\\"branch\\\"},\\\"github_token\\\":{\\\"componentInputParameter\\\":\\\"github_token\\\"},\\\"repo_url\\\":{\\\"componentInputParameter\\\":\\\"repo_url\\\"}}},\\\"taskInfo\\\":{\\\"name\\\":\\\"upload-to-github\\\"}}}},\\\"inputDefinitions\\\":{\\\"parameters\\\":{\\\"branch\\\":{\\\"defaultValue\\\":\\\"report\\\",\\\"isOptional\\\":true,\\\"parameterType\\\":\\\"STRING\\\"},\\\"github_token\\\":{\\\"parameterType\\\":\\\"STRING\\\"},\\\"repo_url\\\":{\\\"parameterType\\\":\\\"STRING\\\"}}}}\"},{\"name\":\"runtime-config\",\"default\":\"\",\"value\":\"{\\\"parameterValues\\\":{\\\"branch\\\":\\\"report\\\",\\\"github_token\\\":\\\"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\\\",\\\"repo_url\\\":\\\"https://github.com/Rosa1026/Kubeflow.git\\\"}}\"},{\"name\":\"task\",\"default\":\"\",\"value\":\"\"},{\"name\":\"parent-dag-id\",\"default\":\"0\",\"value\":\"0\"},{\"name\":\"iteration-index\",\"default\":\"-1\",\"value\":\"-1\"},{\"name\":\"driver-type\",\"default\":\"DAG\",\"value\":\"ROOT_DAG\"}]},\"outputs\":{\"parameters\":[{\"name\":\"execution-id\",\"value\":\"150\",\"valueFrom\":{\"path\":\"/tmp/outputs/execution-id\"}},{\"name\":\"iteration-count\",\"value\":\"0\",\"valueFrom\":{\"path\":\"/tmp/outputs/iteration-count\",\"default\":\"0\"}},{\"name\":\"condition\",\"value\":\"nil\",\"valueFrom\":{\"path\":\"/tmp/outputs/condition\",\"default\":\"true\"}}],\"artifacts\":[{\"name\":\"main-logs\",\"s3\":{\"key\":\"artifacts/pod-log-to-github-gmf4s/2025/05/08/pod-log-to-github-gmf4s-system-dag-driver-2252566396/main.log\"}}],\"exitCode\":\"0\"},\"children\":[\"pod-log-to-github-gmf4s-4161926427\"],\"hostNodeName\":\"k8s-kubeflow-worker01\"},\"pod-log-to-github-gmf4s-3170595587\":{\"id\":\"pod-log-to-github-gmf4s-3170595587\",\"name\":\"pod-log-to-github-gmf4s.root.collect-filtered-logs\",\"displayName\":\"collect-filtered-logs\",\"type\":\"DAG\",\"templateName\":\"system-container-executor\",\"templateScope\":\"local/pod-log-to-github-gmf4s\",\"phase\":\"Running\",\"boundaryID\":\"pod-log-to-github-gmf4s-4161926427\",\"startedAt\":\"2025-05-08T06:36:04Z\",\"finishedAt\":null,\"progress\":\"0/1\",\"inputs\":{\"parameters\":[{\"name\":\"pod-spec-patch\",\"value\":\"{\\\"containers\\\":[{\\\"name\\\":\\\"main\\\",\\\"image\\\":\\\"python:3.9\\\",\\\"command\\\":[\\\"/var/run/argo/argoexec\\\",\\\"emissary\\\",\\\"--\\\",\\\"/kfp-launcher/launch\\\",\\\"--pipeline_name\\\",\\\"pod-log-to-github\\\",\\\"--run_id\\\",\\\"b9842c1b-438d-410c-ad18-f01e0145433b\\\",\\\"--execution_id\\\",\\\"151\\\",\\\"--executor_input\\\",\\\"{\\\\\\\"inputs\\\\\\\":{}, \\\\\\\"outputs\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifacts\\\\\\\":[{\\\\\\\"type\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}, \\\\\\\"uri\\\\\\\":\\\\\\\"minio://mlpipeline/v2/artifacts/pod-log-to-github/b9842c1b-438d-410c-ad18-f01e0145433b/collect-filtered-logs/output_report\\\\\\\"}]}}, \\\\\\\"outputFile\\\\\\\":\\\\\\\"/tmp/kfp_outputs/output_metadata.json\\\\\\\"}}\\\",\\\"--component_spec\\\",\\\"{\\\\\\\"outputDefinitions\\\\\\\":{\\\\\\\"artifacts\\\\\\\":{\\\\\\\"output_report\\\\\\\":{\\\\\\\"artifactType\\\\\\\":{\\\\\\\"schemaTitle\\\\\\\":\\\\\\\"system.Dataset\\\\\\\", \\\\\\\"schemaVersion\\\\\\\":\\\\\\\"0.0.1\\\\\\\"}}}}, \\\\\\\"executorLabel\\\\\\\":\\\\\\\"exec-collect-filtered-logs\\\\\\\"}\\\",\\\"--pod_name\\\",\\\"$(KFP_POD_NAME)\\\",\\\"--pod_uid\\\",\\\"$(KFP_POD_UID)\\\",\\\"--mlmd_server_address\\\",\\\"$(METADATA_GRPC_SERVICE_HOST)\\\",\\\"--mlmd_server_port\\\",\\\"$(METADATA_GRPC_SERVICE_PORT)\\\",\\\"--\\\"],\\\"args\\\":[\\\"sh\\\",\\\"-c\\\",\\\"\\\\nif ! [ -x \\\\\\\"$(command -v pip)\\\\\\\" ]; then\\\\n    python3 -m ensurepip || python3 -m ensurepip --user || apt-get install python3-pip\\\\nfi\\\\n\\\\nPIP_DISABLE_PIP_VERSION_CHECK=1 python3 -m pip install --quiet --no-warn-script-location 'kfp==2.13.0' '--no-deps' 'typing-extensions\\\\u003e=3.7.4,\\\\u003c5; python_version\\\\u003c\\\\\\\"3.9\\\\\\\"' \\\\u0026\\\\u0026 \\\\\\\"$0\\\\\\\" \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"sh\\\",\\\"-ec\\\",\\\"program_path=$(mktemp -d)\\\\n\\\\nprintf \\\\\\\"%s\\\\\\\" \\\\\\\"$0\\\\\\\" \\\\u003e \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"\\\\n_KFP_RUNTIME=true python3 -m kfp.dsl.executor_main                         --component_module_path                         \\\\\\\"$program_path/ephemeral_component.py\\\\\\\"                         \\\\\\\"$@\\\\\\\"\\\\n\\\",\\\"\\\\nimport kfp\\\\nfrom kfp import dsl\\\\nfrom kfp.dsl import *\\\\nfrom typing import *\\\\n\\\\ndef collect_filtered_logs(output_report: Output[Dataset]):\\\\n    import subprocess\\\\n    import os\\\\n\\\\n    tmp_report_path = \\\\\\\"/tmp/report.html\\\\\\\"\\\\n    seen_namespaces = set()\\\\n    namespace_events = {}\\\\n\\\\n    # 주요 키워드 목록\\\\n    describe_keywords = [\\\\\\\"CrashLoopBackOff\\\\\\\", \\\\\\\"OOMKilled\\\\\\\", \\\\\\\"Error\\\\\\\", \\\\\\\"Failed\\\\\\\", \\\\\\\"Back-off\\\\\\\"]\\\\n    log_keywords = [\\\\\\\"Traceback\\\\\\\", \\\\\\\"panic\\\\\\\", \\\\\\\"exception\\\\\\\", \\\\\\\"fail\\\\\\\", \\\\\\\"error\\\\\\\"]\\\\n    event_keywords = [\\\\\\\"Failed\\\\\\\", \\\\\\\"BackOff\\\\\\\", \\\\\\\"Unhealthy\\\\\\\", \\\\\\\"Killing\\\\\\\", \\\\\\\"Evicted\\\\\\\"]\\\\n\\\\n    # kubectl 설치\\\\n    subprocess.run([\\\\\\\"apt-get\\\\\\\", \\\\\\\"update\\\\\\\", \\\\\\\"-y\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\n        \\\\\\\"curl\\\\\\\", \\\\\\\"-LO\\\\\\\", \\\\\\\"https://storage.googleapis.com/kubernetes-release/release/v1.24.0/bin/linux/amd64/kubectl\\\\\\\"\\\\n    ], check=True)\\\\n    subprocess.run([\\\\\\\"chmod\\\\\\\", \\\\\\\"+x\\\\\\\", \\\\\\\"kubectl\\\\\\\"], check=True)\\\\n    subprocess.run([\\\\\\\"mv\\\\\\\", \\\\\\\"kubectl\\\\\\\", \\\\\\\"/usr/local/bin/\\\\\\\"], check=True)\\\\n\\\\n    # Get all pods\\\\n    pods_output = subprocess.getoutput(\\\\\\\"kubectl get pods --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers\\\\\\\")\\\\n    pod_lines = pods_output.strip().split(\\\\\\\"\\\\\\\\n\\\\\\\")\\\\n\\\\n    def match_keywords(lines, keywords):\\\\n        return \\\\\\\"\\\\\\\\n\\\\\\\".join([line for line in lines if any(k in line for k in keywords)])\\\\n\\\\n    with open(tmp_report_path, \\\\\\\"w\\\\\\\") as f:\\\\n        # HTML header\\\\n        f.write(\\\\\\\"\\\\\\\"\\\\\\\"\\\\n        \\\\u003chtml\\\\u003e\\\\n        \\\\u003chead\\\\u003e\\\\n            \\\\u003cmeta charset=\\\\\\\"UTF-8\\\\\\\"\\\\u003e\\\\n            \\\\u003cstyle\\\\u003e\\\\n                body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }\\\\n                h1 { color: #2c3e50; }\\\\n                h2 { color: #34495e; }\\\\n                pre { background-color: #ecf0f1; padding: 10px; border-left: 4px solid #3498db; overflow-x: auto; }\\\\n                section { margin-bottom: 30px; }\\\\n            \\\\u003c/style\\\\u003e\\\\n        \\\\u003c/head\\\\u003e\\\\n        \\\\u003cbody\\\\u003e\\\\n        \\\\u003ch1\\\\u003eKubernetes 장애 점검 리포트\\\\u003c/h1\\\\u003e\\\\n        \\\\\\\"\\\\\\\"\\\\\\\")\\\\n\\\\n        for line in pod_lines:\\\\n            if not line.strip():\\\\n                continue\\\\n            try:\\\\n                namespace, pod_name = line.split()\\\\n            except ValueError:\\\\n                continue\\\\n\\\\n            describe_output = subprocess.getoutput(f\\\\\\\"kubectl describe pod {pod_name} -n {namespace}\\\\\\\")\\\\n            log_output = subprocess.getoutput(f\\\\\\\"kubectl logs {pod_name} -n {namespace}\\\\\\\")\\\\n\\\\n            matched_describe = match_keywords(describe_output.splitlines(), describe_keywords)\\\\n            matched_logs = match_keywords(log_output.splitlines(), log_keywords)\\\\n\\\\n            if matched_describe or matched_logs:\\\\n                f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003ePod: {pod_name} (Namespace: {namespace})\\\\u003c/h2\\\\u003e\\\\\\\")\\\\n                if matched_describe:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eDescribe (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_describe + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                if matched_logs:\\\\n                    f.write(\\\\\\\"\\\\u003ch3\\\\u003eLogs (filtered)\\\\u003c/h3\\\\u003e\\\\u003cpre\\\\u003e\\\\\\\" + matched_logs + \\\\\\\"\\\\u003c/pre\\\\u003e\\\\\\\")\\\\n                f.write(\\\\\\\"\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n            if namespace not in seen_namespaces:\\\\n                events_output = subprocess.getoutput(f\\\\\\\"kubectl get events -n {namespace}\\\\\\\")\\\\n                matched_events = match_keywords(events_output.splitlines(), event_keywords)\\\\n                if matched_events:\\\\n                    namespace_events[namespace] = matched_events\\\\n                seen_namespaces.add(namespace)\\\\n\\\\n        for ns, events in namespace_events.items():\\\\n            f.write(f\\\\\\\"\\\\u003csection\\\\u003e\\\\u003ch2\\\\u003eEvents in Namespace: {ns}\\\\u003c/h2\\\\u003e\\\\u003cpre\\\\u003e{events}\\\\u003c/pre\\\\u003e\\\\u003c/section\\\\u003e\\\\\\\")\\\\n\\\\n        f.write(\\\\\\\"\\\\u003c/body\\\\u003e\\\\u003c/html\\\\u003e\\\\\\\")\\\\n\\\\n    subprocess.run([\\\\\\\"cp\\\\\\\", tmp_report_path, output_report.path], check=True)\\\\n\\\\n\\\",\\\"--executor_input\\\",\\\"{{$}}\\\",\\\"--function_to_execute\\\",\\\"collect_filtered_logs\\\"],\\\"resources\\\":{}}]}\"},{\"name\":\"cached-decision\",\"default\":\"false\",\"value\":\"false\"}]},\"children\":[\"pod-log-to-github-gmf4s-1948391354\"]},\"pod-log-to-github-gmf4s-4161926427\":{\"id\":\"pod-log-to-github-gmf4s-4161926427\",\"name\":\"pod-log-to-github-gmf4s.root\",\"displayName\":\"root\",\"type\":\"DAG\",\"templateName\":\"root\",\"templateScope\":\"local/pod-log-to-github-gmf4s\",\"phase\":\"Running\",\"boundaryID\":\"pod-log-to-github-gmf4s\",\"startedAt\":\"2025-05-08T06:35:54Z\",\"finishedAt\":null,\"progress\":\"1/2\",\"inputs\":{\"parameters\":[{\"name\":\"parent-dag-id\",\"value\":\"150\"}]},\"children\":[\"pod-log-to-github-gmf4s-2119004180\"]}},\"conditions\":[{\"type\":\"PodRunning\",\"status\":\"True\"}],\"resourcesDuration\":{\"cpu\":7,\"memory\":5},\"artifactRepositoryRef\":{\"default\":true,\"artifactRepository\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio-service.kubeflow:9000\",\"bucket\":\"mlpipeline\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"mlpipeline-minio-artifact\",\"key\":\"secretkey\"},\"keyFormat\":\"artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}\"}}},\"artifactGCStatus\":{\"notSpecified\":true}}}: rpc error: code = Unauthenticated desc = Jwt is expired"

Pod: ml-pipeline-scheduledworkflow-658b675548-8pcf8 (Namespace: kubeflow)

Logs (filtered)

W0428 06:44:49.070450       1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.25.9/tools/cache/reflector.go:169: failed to list *v1beta1.ScheduledWorkflow: Get "https://10.96.0.1:443/apis/kubeflow.org/v1beta1/scheduledworkflows?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: connect: connection refused
E0428 06:44:49.070569       1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.25.9/tools/cache/reflector.go:169: Failed to watch *v1beta1.ScheduledWorkflow: failed to list *v1beta1.ScheduledWorkflow: Get "https://10.96.0.1:443/apis/kubeflow.org/v1beta1/scheduledworkflows?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: connect: connection refused
W0428 06:44:49.070653       1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.25.9/tools/cache/reflector.go:169: failed to list *v1alpha1.Workflow: Get "https://10.96.0.1:443/apis/argoproj.io/v1alpha1/workflows?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: connect: connection refused
E0428 06:44:49.070755       1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.25.9/tools/cache/reflector.go:169: Failed to watch *v1alpha1.Workflow: failed to list *v1alpha1.Workflow: Get "https://10.96.0.1:443/apis/argoproj.io/v1alpha1/workflows?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: connect: connection refused

Pod: ml-pipeline-ui-5c66bf88b5-fdzr2 (Namespace: kubeflow)

Logs (filtered)

(node:1) UnhandledPromiseRejectionWarning: FetchError: request to http://metadata/computeMetadata/v1/project/project-id failed, reason: getaddrinfo ENOTFOUND metadata
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 20)
(node:1) UnhandledPromiseRejectionWarning: FetchError: request to http://metadata/computeMetadata/v1/instance/attributes/cluster-name failed, reason: getaddrinfo ENOTFOUND metadata
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 21)
(node:1) UnhandledPromiseRejectionWarning: FetchError: request to http://metadata/computeMetadata/v1/instance/attributes/cluster-name failed, reason: getaddrinfo ENOTFOUND metadata
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 22)
(node:1) UnhandledPromiseRejectionWarning: FetchError: request to http://metadata/computeMetadata/v1/project/project-id failed, reason: getaddrinfo ENOTFOUND metadata
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 23)
(node:1) UnhandledPromiseRejectionWarning: FetchError: request to http://metadata/computeMetadata/v1/project/project-id failed, reason: getaddrinfo ENOTFOUND metadata
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 24)
(node:1) UnhandledPromiseRejectionWarning: FetchError: request to http://metadata/computeMetadata/v1/instance/attributes/cluster-name failed, reason: getaddrinfo ENOTFOUND metadata
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 25)
(node:1) UnhandledPromiseRejectionWarning: FetchError: request to http://metadata/computeMetadata/v1/project/project-id failed, reason: getaddrinfo ENOTFOUND metadata
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 26)
(node:1) UnhandledPromiseRejectionWarning: FetchError: request to http://metadata/computeMetadata/v1/instance/attributes/cluster-name failed, reason: getaddrinfo ENOTFOUND metadata
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 27)
(node:1) UnhandledPromiseRejectionWarning: FetchError: request to http://metadata/computeMetadata/v1/project/project-id failed, reason: getaddrinfo ENOTFOUND metadata
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 28)
(node:1) UnhandledPromiseRejectionWarning: FetchError: request to http://metadata/computeMetadata/v1/instance/attributes/cluster-name failed, reason: getaddrinfo ENOTFOUND metadata
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 29)
(node:1) UnhandledPromiseRejectionWarning: FetchError: request to http://metadata/computeMetadata/v1/instance/attributes/cluster-name failed, reason: getaddrinfo ENOTFOUND metadata
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 30)
(node:1) UnhandledPromiseRejectionWarning: FetchError: request to http://metadata/computeMetadata/v1/project/project-id failed, reason: getaddrinfo ENOTFOUND metadata
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 31)
(node:1) UnhandledPromiseRejectionWarning: FetchError: request to http://metadata/computeMetadata/v1/instance/attributes/cluster-name failed, reason: getaddrinfo ENOTFOUND metadata
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 32)
(node:1) UnhandledPromiseRejectionWarning: FetchError: request to http://metadata/computeMetadata/v1/project/project-id failed, reason: getaddrinfo ENOTFOUND metadata
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 33)

Pod: ml-pipeline-viewer-crd-c7b78c8c-bldlh (Namespace: kubeflow)

Describe (filtered)

      Reason:       Error

Pod: notebook-controller-deployment-5458bf988b-dh2kj (Namespace: kubeflow)

Describe (filtered)

      Reason:       Error

Pod: profiles-deployment-79f5cf977d-6jvns (Namespace: kubeflow)

Describe (filtered)

      Reason:       Error

Pod: pvcviewer-controller-manager-5b8c866b58-qz2hj (Namespace: kubeflow)

Logs (filtered)

E0428 06:49:44.163936       1 leaderelection.go:332] error retrieving resource lock kubeflow/57a72bdf.kubeflow.org: Get "https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kubeflow/leases/57a72bdf.kubeflow.org": dial tcp 10.96.0.1:443: connect: connection refused

Pod: tensorboard-controller-deployment-78f5598f4b-wrbqb (Namespace: kubeflow)

Describe (filtered)

      Reason:       Error

Pod: training-operator-7b99976c6-c2dfm (Namespace: kubeflow)

Logs (filtered)

2025-04-28T06:51:00Z	ERROR	cert-rotation	could not refresh CA and server certs	{"error": "Operation cannot be fulfilled on secrets \"training-operator-webhook-cert\": the object has been modified; please apply your changes to the latest version and try again"}
2025-04-28T06:51:00Z	ERROR	cert-rotation	secret is not well-formed, cannot update webhook configurations	{"error": "Cert secret is not well-formed, missing ca.crt", "errorVerbose": "Cert secret is not well-formed, missing ca.crt\ngithub.com/open-policy-agent/cert-controller/pkg/rotator.buildArtifactsFromSecret\n\t/go/pkg/mod/github.com/open-policy-agent/cert-controller@v0.10.1/pkg/rotator/rotator.go:501\ngithub.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).Reconcile\n\t/go/pkg/mod/github.com/open-policy-agent/cert-controller@v0.10.1/pkg/rotator/rotator.go:778\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}

Pod: workflow-controller-68d76bcb8b-d4nps (Namespace: kubeflow)

Describe (filtered)

      Reason:       Error

Logs (filtered)

time="2025-04-29T01:06:47.140Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-07T02:25:15.237Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-07T02:26:25.726Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=review-pipeline-ph7jk-system-container-impl-1764901853 templateName=system-container-impl workflow=review-pipeline-ph7jk
time="2025-05-07T02:30:15.799Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-07T02:31:06.108Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=review-pipeline-9wl4l-system-container-impl-2576199031 templateName=system-container-impl workflow=review-pipeline-9wl4l
time="2025-05-07T04:26:15.590Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-07T04:27:05.812Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=review-pipeline-2hczt-system-container-impl-257074056 templateName=system-container-impl workflow=review-pipeline-2hczt
time="2025-05-07T04:33:26.618Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-07T04:34:17.581Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=review-pipeline-w64f8-system-container-impl-2840241048 templateName=system-container-impl workflow=review-pipeline-w64f8
time="2025-05-07T04:36:32.190Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-07T04:37:23.381Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=review-pipeline-m8w8c-system-container-impl-2662602218 templateName=system-container-impl workflow=review-pipeline-m8w8c
time="2025-05-07T05:11:46.278Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-07T05:12:56.723Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-2s5r9-system-container-impl-2521372343 templateName=system-container-impl workflow=pod-log-to-github-2s5r9
time="2025-05-07T05:23:19.536Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-07T05:24:09.078Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-07T05:24:59.744Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-r464q-system-container-impl-2023972999 templateName=system-container-impl workflow=pod-log-to-github-r464q
time="2025-05-07T05:25:24.131Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-07T05:26:14.395Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-t4w5w-system-container-impl-2477464621 templateName=system-container-impl workflow=pod-log-to-github-t4w5w
time="2025-05-07T05:31:20.990Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-07T05:32:11.914Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-xkz8p-system-container-impl-1495302353 templateName=system-container-impl workflow=pod-log-to-github-xkz8p
time="2025-05-07T05:36:04.555Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-07T05:37:15.673Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-2tkgb-system-container-impl-819368644 templateName=system-container-impl workflow=pod-log-to-github-2tkgb
time="2025-05-07T05:43:53.353Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-07T05:44:47.779Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-qv4f5-system-container-impl-742416059 templateName=system-container-impl workflow=pod-log-to-github-qv4f5
time="2025-05-07T05:49:01.019Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-07T05:49:31.193Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-gfhgz-system-container-impl-3552771745 templateName=system-container-impl workflow=pod-log-to-github-gfhgz
time="2025-05-07T05:52:42.346Z" level=warning msg="Coudn't obtain child for pod-log-to-github-gfhgz-71102127, panicking"
time="2025-05-07T05:52:42.346Z" level=warning msg="Coudn't obtain child for pod-log-to-github-gfhgz-3552771745, panicking"
time="2025-05-07T05:56:41.977Z" level=warning msg="Coudn't obtain child for pod-log-to-github-gfhgz-71102127, panicking"
time="2025-05-07T05:56:41.977Z" level=warning msg="Coudn't obtain child for pod-log-to-github-gfhgz-3552771745, panicking"
time="2025-05-07T06:54:13.593Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-07T06:54:43.764Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-7xk8h-system-container-impl-2399659743 templateName=system-container-impl workflow=pod-log-to-github-7xk8h
time="2025-05-07T07:34:04.979Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-07T07:34:35.143Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-6lxcf-system-container-impl-508915066 templateName=system-container-impl workflow=pod-log-to-github-6lxcf
time="2025-05-07T07:35:54.714Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-07T07:36:24.846Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-vpkhs-system-container-impl-2059144489 templateName=system-container-impl workflow=pod-log-to-github-vpkhs
time="2025-05-07T07:37:22.544Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-07T07:40:40.005Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-07T07:41:20.489Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-qlsrg-system-container-impl-2629519936 templateName=system-container-impl workflow=pod-log-to-github-qlsrg
time="2025-05-07T07:43:08.355Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-07T07:44:20.945Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-2lp4x-system-container-impl-424932916 templateName=system-container-impl workflow=pod-log-to-github-2lp4x
time="2025-05-07T07:49:44.525Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-07T07:50:36.355Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-cdd4s-system-container-impl-4284018394 templateName=system-container-impl workflow=pod-log-to-github-cdd4s
time="2025-05-07T07:52:56.292Z" level=info msg="signaled container" container=wait error="" namespace=kubeflow-user-example-com pod=pod-log-to-github-z9lb2-system-container-impl-2128329098 stderr= stdout="killing 1 with terminated\n"
time="2025-05-07T07:53:18.821Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-07T07:56:19.971Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-07T07:57:03.966Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-vc5rx-system-container-impl-1672389891 templateName=system-container-impl workflow=pod-log-to-github-vc5rx
time="2025-05-07T07:58:50.667Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-07T08:04:46.620Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-07T08:04:56.675Z" level=info msg="Pod failed: Error (exit code 1)" displayName=root-driver namespace=kubeflow-user-example-com pod=pod-log-to-github-dnplr-system-dag-driver-1081258443 templateName=system-dag-driver workflow=pod-log-to-github-dnplr
time="2025-05-07T08:08:43.342Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-07T08:13:53.241Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-07T08:18:45.707Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-07T08:20:28.475Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-07T08:42:16.833Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-07T08:45:03.823Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-07T08:51:59.337Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-08T00:31:27.393Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-08T00:32:17.839Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-vkpgx-system-container-impl-1923254719 templateName=system-container-impl workflow=pod-log-to-github-vkpgx
time="2025-05-08T00:33:57.622Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-08T00:35:50.836Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-sqv6q-system-container-impl-41613183 templateName=system-container-impl workflow=pod-log-to-github-sqv6q
time="2025-05-08T00:40:52.932Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-08T00:42:50.397Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-lq7m2-system-container-impl-1051008407 templateName=system-container-impl workflow=pod-log-to-github-lq7m2
time="2025-05-08T00:44:23.373Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-08T00:46:25.659Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-nqs86-system-container-impl-3731481970 templateName=system-container-impl workflow=pod-log-to-github-nqs86
time="2025-05-08T01:01:54.875Z" level=warning msg="Coudn't obtain child for pod-log-to-github-nqs86-2763467937, panicking"
time="2025-05-08T01:01:54.875Z" level=warning msg="Coudn't obtain child for pod-log-to-github-nqs86-331213999, panicking"
time="2025-05-08T01:01:54.876Z" level=warning msg="Coudn't obtain child for pod-log-to-github-nqs86-3731481970, panicking"
time="2025-05-08T01:03:32.667Z" level=warning msg="Coudn't obtain child for pod-log-to-github-nqs86-2763467937, panicking"
time="2025-05-08T01:03:32.667Z" level=warning msg="Coudn't obtain child for pod-log-to-github-nqs86-331213999, panicking"
time="2025-05-08T01:03:32.667Z" level=warning msg="Coudn't obtain child for pod-log-to-github-nqs86-3731481970, panicking"
time="2025-05-08T01:03:52.153Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-08T01:05:56.265Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-kcp9j-system-container-impl-1592955769 templateName=system-container-impl workflow=pod-log-to-github-kcp9j
time="2025-05-08T01:09:34.677Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-08T01:11:43.651Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-lm5jl-system-container-impl-3089604882 templateName=system-container-impl workflow=pod-log-to-github-lm5jl
time="2025-05-08T01:50:00.582Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-08T01:52:04.109Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-5sf2x-system-container-impl-2717635514 templateName=system-container-impl workflow=pod-log-to-github-5sf2x
time="2025-05-08T01:52:37.277Z" level=warning msg="Coudn't obtain child for pod-log-to-github-5sf2x-2717635514, panicking"
time="2025-05-08T01:52:37.277Z" level=warning msg="Coudn't obtain child for pod-log-to-github-5sf2x-3759230217, panicking"
time="2025-05-08T01:52:37.277Z" level=warning msg="Coudn't obtain child for pod-log-to-github-5sf2x-4249229351, panicking"
time="2025-05-08T01:56:46.362Z" level=warning msg="Coudn't obtain child for pod-log-to-github-5sf2x-2717635514, panicking"
time="2025-05-08T01:56:46.362Z" level=warning msg="Coudn't obtain child for pod-log-to-github-5sf2x-4249229351, panicking"
time="2025-05-08T01:56:46.362Z" level=warning msg="Coudn't obtain child for pod-log-to-github-5sf2x-3759230217, panicking"
time="2025-05-08T02:00:03.805Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-08T04:28:48.343Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-08T04:46:15.082Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-08T04:46:55.702Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-85dff-system-container-impl-1792978204 templateName=system-container-impl workflow=pod-log-to-github-85dff
time="2025-05-08T04:51:41.895Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-08T04:53:54.107Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-phkw4-system-container-impl-1729468608 templateName=system-container-impl workflow=pod-log-to-github-phkw4
time="2025-05-08T04:54:51.648Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-08T04:57:06.241Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-mml78-system-container-impl-4210283497 templateName=system-container-impl workflow=pod-log-to-github-mml78
time="2025-05-08T05:02:59.515Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-08T05:05:17.422Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-zctnf-system-container-impl-3813999009 templateName=system-container-impl workflow=pod-log-to-github-zctnf
time="2025-05-08T05:05:30.241Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-08T05:07:48.139Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-ljqp8-system-container-impl-4244329241 templateName=system-container-impl workflow=pod-log-to-github-ljqp8
time="2025-05-08T05:25:52.238Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-08T05:28:11.176Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-8mjdq-system-container-impl-3832912982 templateName=system-container-impl workflow=pod-log-to-github-8mjdq
time="2025-05-08T05:41:06.421Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-08T05:43:24.902Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-zj76w-system-container-impl-1065118244 templateName=system-container-impl workflow=pod-log-to-github-zj76w
time="2025-05-08T05:50:53.674Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-08T05:53:10.349Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-scgtw-system-container-impl-2117547052 templateName=system-container-impl workflow=pod-log-to-github-scgtw
time="2025-05-08T05:54:29.637Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-08T05:56:47.774Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-pm5rh-system-container-impl-3177957402 templateName=system-container-impl workflow=pod-log-to-github-pm5rh
time="2025-05-08T05:57:49.000Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-08T06:00:11.208Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-tvwg6-system-container-impl-1502162644 templateName=system-container-impl workflow=pod-log-to-github-tvwg6
time="2025-05-08T06:00:56.622Z" level=warning msg="Coudn't obtain child for pod-log-to-github-tvwg6-3280166619, panicking"
time="2025-05-08T06:00:56.622Z" level=warning msg="Coudn't obtain child for pod-log-to-github-tvwg6-1487588749, panicking"
time="2025-05-08T06:00:56.622Z" level=warning msg="Coudn't obtain child for pod-log-to-github-tvwg6-1502162644, panicking"
time="2025-05-08T06:04:16.704Z" level=warning msg="Coudn't obtain child for pod-log-to-github-tvwg6-1487588749, panicking"
time="2025-05-08T06:04:16.704Z" level=warning msg="Coudn't obtain child for pod-log-to-github-tvwg6-3280166619, panicking"
time="2025-05-08T06:04:16.704Z" level=warning msg="Coudn't obtain child for pod-log-to-github-tvwg6-1502162644, panicking"
time="2025-05-08T06:04:37.158Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-08T06:12:46.441Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-08T06:15:04.559Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-nm76f-system-container-impl-3124929480 templateName=system-container-impl workflow=pod-log-to-github-nm76f
time="2025-05-08T06:17:52.073Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-08T06:28:52.368Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-08T06:28:57.641Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-08T06:31:42.218Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2025-05-08T06:34:06.690Z" level=info msg="Pod failed: Error (exit code 1)" displayName=executor namespace=kubeflow-user-example-com pod=pod-log-to-github-z2xx6-system-container-impl-1861443604 templateName=system-container-impl workflow=pod-log-to-github-z2xx6
time="2025-05-08T06:35:44.399Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"

Pod: local-path-provisioner-6548cc785f-xnkv4 (Namespace: local-path-storage)

Logs (filtered)

time="2025-04-25T02:25:42Z" level=error msg="failed to save helper-pod-create-pvc-7305ed53-e3bc-4084-9605-5e80f3e1edd1 logs: error in opening stream: container \"helper-pod\" in pod \"helper-pod-create-pvc-7305ed53-e3bc-4084-9605-5e80f3e1edd1\" is waiting to start: ContainerCreating" 
I0425 02:25:42.092315       1 controller.go:954] "Retrying syncing claim because failures < threshold" key="7305ed53-e3bc-4084-9605-5e80f3e1edd1" failures=0 threshold=15
E0425 02:25:42.092390       1 controller.go:974] "Unhandled Error" err="error syncing claim \"7305ed53-e3bc-4084-9605-5e80f3e1edd1\": failed to provision volume with StorageClass \"local-path\": failed to create volume pvc-7305ed53-e3bc-4084-9605-5e80f3e1edd1: create process timeout after 120 seconds" logger="UnhandledError"
I0425 02:25:42.092434       1 event.go:389] "Event occurred" object="kubeflow/minio-pvc" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Warning" reason="ProvisioningFailed" message="failed to provision volume with StorageClass \"local-path\": failed to create volume pvc-7305ed53-e3bc-4084-9605-5e80f3e1edd1: create process timeout after 120 seconds"
time="2025-04-25T02:25:43Z" level=error msg="failed to save helper-pod-create-pvc-2fb0780e-ff15-49e7-94c2-fbef191fe778 logs: error in opening stream: container \"helper-pod\" in pod \"helper-pod-create-pvc-2fb0780e-ff15-49e7-94c2-fbef191fe778\" is waiting to start: ContainerCreating" 
I0425 02:25:43.733770       1 controller.go:954] "Retrying syncing claim because failures < threshold" key="2fb0780e-ff15-49e7-94c2-fbef191fe778" failures=0 threshold=15
E0425 02:25:43.733820       1 controller.go:974] "Unhandled Error" err="error syncing claim \"2fb0780e-ff15-49e7-94c2-fbef191fe778\": failed to provision volume with StorageClass \"local-path\": failed to create volume pvc-2fb0780e-ff15-49e7-94c2-fbef191fe778: create process timeout after 120 seconds" logger="UnhandledError"
I0425 02:25:43.733918       1 event.go:389] "Event occurred" object="kubeflow/katib-mysql" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Warning" reason="ProvisioningFailed" message="failed to provision volume with StorageClass \"local-path\": failed to create volume pvc-2fb0780e-ff15-49e7-94c2-fbef191fe778: create process timeout after 120 seconds"
time="2025-04-25T02:27:01Z" level=error msg="failed to save  logs: error in opening stream: resource name may not be empty" 
time="2025-04-25T02:27:01Z" level=error msg="unable to delete the helper pod: pods \"helper-pod-create-pvc-7305ed53-e3bc-4084-9605-5e80f3e1edd1\" not found" 
I0425 02:27:01.313372       1 controller.go:954] "Retrying syncing claim because failures < threshold" key="7305ed53-e3bc-4084-9605-5e80f3e1edd1" failures=1 threshold=15
E0425 02:27:01.313410       1 controller.go:974] "Unhandled Error" err="error syncing claim \"7305ed53-e3bc-4084-9605-5e80f3e1edd1\": failed to provision volume with StorageClass \"local-path\": failed to create volume pvc-7305ed53-e3bc-4084-9605-5e80f3e1edd1: pods \"helper-pod-create-pvc-7305ed53-e3bc-4084-9605-5e80f3e1edd1\" not found" logger="UnhandledError"
I0425 02:27:01.313528       1 event.go:389] "Event occurred" object="kubeflow/minio-pvc" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Warning" reason="ProvisioningFailed" message="failed to provision volume with StorageClass \"local-path\": failed to create volume pvc-7305ed53-e3bc-4084-9605-5e80f3e1edd1: pods \"helper-pod-create-pvc-7305ed53-e3bc-4084-9605-5e80f3e1edd1\" not found"
time="2025-04-25T02:27:59Z" level=error msg="failed to save  logs: error in opening stream: resource name may not be empty" 
I0425 02:27:59.092769       1 controller.go:954] "Retrying syncing claim because failures < threshold" key="2fb0780e-ff15-49e7-94c2-fbef191fe778" failures=1 threshold=15
E0425 02:27:59.092827       1 controller.go:974] "Unhandled Error" err="error syncing claim \"2fb0780e-ff15-49e7-94c2-fbef191fe778\": failed to provision volume with StorageClass \"local-path\": failed to create volume pvc-2fb0780e-ff15-49e7-94c2-fbef191fe778: create process timeout after 120 seconds" logger="UnhandledError"
I0425 02:27:59.092875       1 event.go:389] "Event occurred" object="kubeflow/katib-mysql" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Warning" reason="ProvisioningFailed" message="failed to provision volume with StorageClass \"local-path\": failed to create volume pvc-2fb0780e-ff15-49e7-94c2-fbef191fe778: create process timeout after 120 seconds"
W0425 06:37:41.240795       1 reflector.go:492] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:861: watch of *v1.StorageClass ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0425 06:37:41.240795       1 reflector.go:492] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:858: watch of *v1.PersistentVolume ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0425 06:37:41.240795       1 reflector.go:492] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:855: watch of *v1.PersistentVolumeClaim ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding
W0425 06:37:52.533927       1 reflector.go:569] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:858: failed to list *v1.PersistentVolume: Get "https://10.96.0.1:443/api/v1/persistentvolumes?resourceVersion=90296": net/http: TLS handshake timeout
E0425 06:37:52.534010       1 reflector.go:166] "Unhandled Error" err="sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:858: Failed to watch *v1.PersistentVolume: failed to list *v1.PersistentVolume: Get \"https://10.96.0.1:443/api/v1/persistentvolumes?resourceVersion=90296\": net/http: TLS handshake timeout" logger="UnhandledError"
W0425 06:37:52.689638       1 reflector.go:569] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:861: failed to list *v1.StorageClass: Get "https://10.96.0.1:443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=90285": net/http: TLS handshake timeout
E0425 06:37:52.689686       1 reflector.go:166] "Unhandled Error" err="sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:861: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: Get \"https://10.96.0.1:443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=90285\": net/http: TLS handshake timeout" logger="UnhandledError"
W0425 06:37:52.765329       1 reflector.go:569] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:855: failed to list *v1.PersistentVolumeClaim: Get "https://10.96.0.1:443/api/v1/persistentvolumeclaims?resourceVersion=90284": net/http: TLS handshake timeout
E0425 06:37:52.765439       1 reflector.go:166] "Unhandled Error" err="sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:855: Failed to watch *v1.PersistentVolumeClaim: failed to list *v1.PersistentVolumeClaim: Get \"https://10.96.0.1:443/api/v1/persistentvolumeclaims?resourceVersion=90284\": net/http: TLS handshake timeout" logger="UnhandledError"
W0425 06:38:04.318385       1 reflector.go:569] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:861: failed to list *v1.StorageClass: Get "https://10.96.0.1:443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=90285": net/http: TLS handshake timeout
E0425 06:38:04.318452       1 reflector.go:166] "Unhandled Error" err="sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:861: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: Get \"https://10.96.0.1:443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=90285\": net/http: TLS handshake timeout" logger="UnhandledError"
W0425 06:38:05.529148       1 reflector.go:569] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:858: failed to list *v1.PersistentVolume: Get "https://10.96.0.1:443/api/v1/persistentvolumes?resourceVersion=90296": net/http: TLS handshake timeout
E0425 06:38:05.529205       1 reflector.go:166] "Unhandled Error" err="sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:858: Failed to watch *v1.PersistentVolume: failed to list *v1.PersistentVolume: Get \"https://10.96.0.1:443/api/v1/persistentvolumes?resourceVersion=90296\": net/http: TLS handshake timeout" logger="UnhandledError"
W0425 06:38:05.721892       1 reflector.go:569] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:855: failed to list *v1.PersistentVolumeClaim: Get "https://10.96.0.1:443/api/v1/persistentvolumeclaims?resourceVersion=90284": net/http: TLS handshake timeout
E0425 06:38:05.721974       1 reflector.go:166] "Unhandled Error" err="sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:855: Failed to watch *v1.PersistentVolumeClaim: failed to list *v1.PersistentVolumeClaim: Get \"https://10.96.0.1:443/api/v1/persistentvolumeclaims?resourceVersion=90284\": net/http: TLS handshake timeout" logger="UnhandledError"
W0425 06:38:19.242385       1 reflector.go:569] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:861: failed to list *v1.StorageClass: Get "https://10.96.0.1:443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=90285": net/http: TLS handshake timeout
E0425 06:38:19.242487       1 reflector.go:166] "Unhandled Error" err="sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:861: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: Get \"https://10.96.0.1:443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=90285\": net/http: TLS handshake timeout" logger="UnhandledError"
W0425 06:38:19.947447       1 reflector.go:569] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:855: failed to list *v1.PersistentVolumeClaim: Get "https://10.96.0.1:443/api/v1/persistentvolumeclaims?resourceVersion=90284": net/http: TLS handshake timeout
E0425 06:38:19.947503       1 reflector.go:166] "Unhandled Error" err="sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:855: Failed to watch *v1.PersistentVolumeClaim: failed to list *v1.PersistentVolumeClaim: Get \"https://10.96.0.1:443/api/v1/persistentvolumeclaims?resourceVersion=90284\": net/http: TLS handshake timeout" logger="UnhandledError"
W0425 06:38:21.578688       1 reflector.go:569] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:858: failed to list *v1.PersistentVolume: Get "https://10.96.0.1:443/api/v1/persistentvolumes?resourceVersion=90296": net/http: TLS handshake timeout
E0425 06:38:21.578767       1 reflector.go:166] "Unhandled Error" err="sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:858: Failed to watch *v1.PersistentVolume: failed to list *v1.PersistentVolume: Get \"https://10.96.0.1:443/api/v1/persistentvolumes?resourceVersion=90296\": net/http: TLS handshake timeout" logger="UnhandledError"
W0425 06:38:37.891570       1 reflector.go:569] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:861: failed to list *v1.StorageClass: Get "https://10.96.0.1:443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=90285": net/http: TLS handshake timeout
E0425 06:38:37.891630       1 reflector.go:166] "Unhandled Error" err="sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:861: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: Get \"https://10.96.0.1:443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=90285\": net/http: TLS handshake timeout" logger="UnhandledError"
W0425 06:38:42.723949       1 reflector.go:569] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:855: failed to list *v1.PersistentVolumeClaim: Get "https://10.96.0.1:443/api/v1/persistentvolumeclaims?resourceVersion=90284": net/http: TLS handshake timeout
E0425 06:38:42.724009       1 reflector.go:166] "Unhandled Error" err="sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:855: Failed to watch *v1.PersistentVolumeClaim: failed to list *v1.PersistentVolumeClaim: Get \"https://10.96.0.1:443/api/v1/persistentvolumeclaims?resourceVersion=90284\": net/http: TLS handshake timeout" logger="UnhandledError"
W0425 06:38:43.580460       1 reflector.go:569] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:858: failed to list *v1.PersistentVolume: Get "https://10.96.0.1:443/api/v1/persistentvolumes?resourceVersion=90296": net/http: TLS handshake timeout
E0425 06:38:43.580519       1 reflector.go:166] "Unhandled Error" err="sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:858: Failed to watch *v1.PersistentVolume: failed to list *v1.PersistentVolume: Get \"https://10.96.0.1:443/api/v1/persistentvolumes?resourceVersion=90296\": net/http: TLS handshake timeout" logger="UnhandledError"
W0425 06:39:01.049337       1 reflector.go:569] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:861: failed to list *v1.StorageClass: Get "https://10.96.0.1:443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=90285": net/http: TLS handshake timeout
E0425 06:39:01.049392       1 reflector.go:166] "Unhandled Error" err="sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:861: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: Get \"https://10.96.0.1:443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=90285\": net/http: TLS handshake timeout" logger="UnhandledError"
W0425 06:39:10.245987       1 reflector.go:569] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:855: failed to list *v1.PersistentVolumeClaim: Get "https://10.96.0.1:443/api/v1/persistentvolumeclaims?resourceVersion=90284": net/http: TLS handshake timeout
E0425 06:39:10.246061       1 reflector.go:166] "Unhandled Error" err="sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:855: Failed to watch *v1.PersistentVolumeClaim: failed to list *v1.PersistentVolumeClaim: Get \"https://10.96.0.1:443/api/v1/persistentvolumeclaims?resourceVersion=90284\": net/http: TLS handshake timeout" logger="UnhandledError"
W0425 06:39:15.175476       1 reflector.go:569] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:858: failed to list *v1.PersistentVolume: Get "https://10.96.0.1:443/api/v1/persistentvolumes?resourceVersion=90296": net/http: TLS handshake timeout
E0425 06:39:15.175535       1 reflector.go:166] "Unhandled Error" err="sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:858: Failed to watch *v1.PersistentVolume: failed to list *v1.PersistentVolume: Get \"https://10.96.0.1:443/api/v1/persistentvolumes?resourceVersion=90296\": net/http: TLS handshake timeout" logger="UnhandledError"
W0425 06:39:53.060968       1 reflector.go:569] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:861: failed to list *v1.StorageClass: Get "https://10.96.0.1:443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=90285": net/http: TLS handshake timeout
E0425 06:39:53.061062       1 reflector.go:166] "Unhandled Error" err="sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:861: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: Get \"https://10.96.0.1:443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=90285\": net/http: TLS handshake timeout" logger="UnhandledError"
W0425 06:39:59.505756       1 reflector.go:569] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:858: failed to list *v1.PersistentVolume: Get "https://10.96.0.1:443/api/v1/persistentvolumes?resourceVersion=90296": net/http: TLS handshake timeout
E0425 06:39:59.505845       1 reflector.go:166] "Unhandled Error" err="sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:858: Failed to watch *v1.PersistentVolume: failed to list *v1.PersistentVolume: Get \"https://10.96.0.1:443/api/v1/persistentvolumes?resourceVersion=90296\": net/http: TLS handshake timeout" logger="UnhandledError"
W0425 06:40:08.310477       1 reflector.go:569] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:855: failed to list *v1.PersistentVolumeClaim: Get "https://10.96.0.1:443/api/v1/persistentvolumeclaims?resourceVersion=90284": net/http: TLS handshake timeout
E0425 06:40:08.310544       1 reflector.go:166] "Unhandled Error" err="sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:855: Failed to watch *v1.PersistentVolumeClaim: failed to list *v1.PersistentVolumeClaim: Get \"https://10.96.0.1:443/api/v1/persistentvolumeclaims?resourceVersion=90284\": net/http: TLS handshake timeout" logger="UnhandledError"
W0425 06:40:56.492490       1 reflector.go:569] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:861: failed to list *v1.StorageClass: Get "https://10.96.0.1:443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=90285": net/http: TLS handshake timeout
E0425 06:40:56.492552       1 reflector.go:166] "Unhandled Error" err="sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:861: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: Get \"https://10.96.0.1:443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=90285\": net/http: TLS handshake timeout" logger="UnhandledError"
W0425 06:41:01.429178       1 reflector.go:569] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:858: failed to list *v1.PersistentVolume: Get "https://10.96.0.1:443/api/v1/persistentvolumes?resourceVersion=90296": net/http: TLS handshake timeout
E0425 06:41:01.429235       1 reflector.go:166] "Unhandled Error" err="sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:858: Failed to watch *v1.PersistentVolume: failed to list *v1.PersistentVolume: Get \"https://10.96.0.1:443/api/v1/persistentvolumes?resourceVersion=90296\": net/http: TLS handshake timeout" logger="UnhandledError"
W0425 06:41:15.486683       1 reflector.go:569] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:855: failed to list *v1.PersistentVolumeClaim: Get "https://10.96.0.1:443/api/v1/persistentvolumeclaims?resourceVersion=90284": net/http: TLS handshake timeout
E0425 06:41:15.486744       1 reflector.go:166] "Unhandled Error" err="sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:855: Failed to watch *v1.PersistentVolumeClaim: failed to list *v1.PersistentVolumeClaim: Get \"https://10.96.0.1:443/api/v1/persistentvolumeclaims?resourceVersion=90284\": net/http: TLS handshake timeout" logger="UnhandledError"
W0425 06:41:43.637806       1 reflector.go:569] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:861: failed to list *v1.StorageClass: Get "https://10.96.0.1:443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=90285": net/http: TLS handshake timeout
E0425 06:41:43.637891       1 reflector.go:166] "Unhandled Error" err="sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:861: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: Get \"https://10.96.0.1:443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=90285\": net/http: TLS handshake timeout" logger="UnhandledError"
W0425 06:42:09.171720       1 reflector.go:569] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:858: failed to list *v1.PersistentVolume: Get "https://10.96.0.1:443/api/v1/persistentvolumes?resourceVersion=90296": net/http: TLS handshake timeout
E0425 06:42:09.171803       1 reflector.go:166] "Unhandled Error" err="sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:858: Failed to watch *v1.PersistentVolume: failed to list *v1.PersistentVolume: Get \"https://10.96.0.1:443/api/v1/persistentvolumes?resourceVersion=90296\": net/http: TLS handshake timeout" logger="UnhandledError"
W0425 06:42:17.327645       1 reflector.go:569] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:855: failed to list *v1.PersistentVolumeClaim: Get "https://10.96.0.1:443/api/v1/persistentvolumeclaims?resourceVersion=90284": net/http: TLS handshake timeout
E0425 06:42:17.327705       1 reflector.go:166] "Unhandled Error" err="sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:855: Failed to watch *v1.PersistentVolumeClaim: failed to list *v1.PersistentVolumeClaim: Get \"https://10.96.0.1:443/api/v1/persistentvolumeclaims?resourceVersion=90284\": net/http: TLS handshake timeout" logger="UnhandledError"
E0425 07:08:56.939803       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:09:23.498108       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:12:30.543793       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:16:38.943288       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:17:21.500838       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:18:53.546269       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:26:08.946017       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:26:38.503159       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:28:42.549106       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:33:12.507215       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:34:22.949061       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:35:13.551121       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:39:25.510387       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:40:22.952394       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:43:32.554758       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:44:34.512606       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:46:34.956084       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:49:51.557484       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:53:49.515290       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:54:54.958097       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:59:05.560671       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 07:59:37.517229       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:00:35.960322       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:05:40.519773       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:07:30.563037       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:09:17.962320       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:14:11.521695       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:14:56.565826       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:18:12.964425       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:19:26.524517       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:24:09.567892       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:25:31.526861       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:27:34.966322       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:30:58.570872       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:33:22.969008       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:35:08.529310       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:39:26.573588       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:41:19.531018       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:43:18.971249       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:44:48.576182       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:46:21.533809       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:49:51.578547       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:52:15.536999       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:52:17.973187       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:57:31.581642       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 08:57:55.975549       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:01:59.540013       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:03:03.584265       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:06:50.977919       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:07:31.542678       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:11:24.588557       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:14:40.545404       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:15:28.980236       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:17:07.591200       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:22:12.547714       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:25:07.984057       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:26:31.594067       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:28:09.550351       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:32:05.986448       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:32:46.596244       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:34:54.552167       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:37:11.988412       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:38:34.598486       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:42:16.555300       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:43:50.991186       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:47:49.601467       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:51:31.557245       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:53:35.993068       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 09:53:53.604068       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:01:05.996092       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:01:15.559102       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:02:38.606856       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:08:07.561278       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:09:49.609889       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:10:26.998337       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:15:33.000785       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:17:19.563519       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:18:09.611827       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:23:48.003252       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:24:19.614989       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:24:58.566086       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:32:58.568973       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:33:18.005578       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:33:47.631229       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:38:37.571520       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:41:05.644707       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:43:04.008073       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:45:57.573289       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:48:47.700513       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:50:14.009804       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:55:18.576486       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:56:52.709215       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 10:57:44.011885       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:03:12.578893       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:04:39.014725       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:06:43.714852       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:10:08.581223       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:11:07.018002       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:14:29.717132       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:18:24.021899       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:18:49.582844       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:21:07.719187       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:25:24.023997       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:25:27.585262       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:27:46.721034       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:33:23.027607       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:35:12.586992       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:37:03.723347       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:41:30.029717       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:42:24.589375       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:45:26.725069       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:49:24.031720       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:50:49.727036       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:51:49.591642       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:56:57.034795       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:57:00.593986       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 11:57:35.729155       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:03:42.037706       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:05:14.731259       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:05:40.595870       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:11:25.733391       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:12:00.040541       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:13:42.597716       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:18:02.735578       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:20:55.043246       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:22:44.600325       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:24:17.738324       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:28:42.045550       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:30:12.602443       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:30:44.740381       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:34:42.048345       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:37:27.606676       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:40:21.742370       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:40:56.050366       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:43:11.608859       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:45:42.744765       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:48:41.052207       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:51:25.612281       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:53:24.746676       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:54:03.055149       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 12:58:21.614430       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:01:51.057957       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:02:40.748853       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:06:32.617180       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:08:10.060082       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:08:24.751525       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:14:35.062311       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:15:09.619681       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:16:59.753310       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:21:07.065148       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:22:46.756135       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:23:30.621667       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:29:20.068211       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:30:35.759446       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:31:41.623407       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:34:48.071064       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:37:32.761416       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:40:24.073547       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:41:39.625623       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:44:22.763083       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:47:51.627962       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:49:08.075288       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:51:31.765410       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:56:32.077824       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:56:54.630776       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 13:59:17.767648       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:02:07.633255       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:05:33.079860       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:08:11.636057       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:08:55.770366       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:13:45.082191       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:15:41.773842       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:17:33.639125       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:22:02.085020       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:23:03.776491       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:27:23.087306       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:27:32.641121       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:29:23.778367       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:32:45.644253       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:36:16.090240       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:36:26.781157       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:39:57.646294       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:44:41.093032       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:45:54.648729       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:46:02.783226       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:52:35.095568       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:53:43.785259       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 14:55:41.650677       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:01:50.098003       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:02:02.788327       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:03:55.652657       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:11:11.100561       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:11:36.792063       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:12:13.655127       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:18:07.657664       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:19:46.796368       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:20:44.103006       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:25:22.659615       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:25:23.799228       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:29:52.105316       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:33:02.801784       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:33:07.662221       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:35:04.107083       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:38:38.664182       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:42:18.803990       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:44:55.109789       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:46:45.666135       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:50:19.805978       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:51:09.112726       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:55:27.668572       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:59:48.808273       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 15:59:54.115206       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:05:00.671636       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:05:38.117278       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:05:51.811390       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:12:27.674215       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:14:33.120181       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:14:34.820213       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:19:10.676877       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:24:09.822980       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:24:22.678518       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:24:23.122640       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:32:25.125829       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:32:35.680520       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:33:03.825843       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:40:49.128797       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:40:53.827954       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:41:25.683028       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:47:21.131252       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:48:58.685416       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:50:44.830185       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:54:51.687767       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:54:59.133981       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 16:58:57.832334       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:00:21.689777       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:02:06.137058       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:05:20.835084       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:08:31.691624       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:09:25.139134       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:10:25.838128       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:14:51.693946       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:16:37.141525       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:17:28.840988       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:21:58.695927       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:22:41.842895       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:26:14.144490       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:30:14.697781       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:32:07.845149       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:34:54.147083       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:36:44.699323       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:40:25.847959       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:40:39.149638       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:45:43.707138       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:47:38.851214       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:50:11.151877       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:53:28.710503       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:54:57.855405       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 17:56:59.154730       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:02:00.712487       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:03:50.859059       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:06:34.157379       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:11:31.861607       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:11:34.714836       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:13:03.159808       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:17:26.717176       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:18:06.864638       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:21:32.162301       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:22:51.719072       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:24:56.867414       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:29:13.164265       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:29:52.721897       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:30:47.875253       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:35:45.724137       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:37:00.166492       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:39:21.877163       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:44:16.169936       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:45:19.726530       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:48:31.879662       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:50:45.172329       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:51:57.728072       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:56:26.881714       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 18:57:18.174614       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:01:40.730010       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:03:52.883883       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:05:55.176791       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:10:34.886146       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:10:46.732031       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:12:14.179013       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:16:17.735478       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:16:27.887943       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:18:00.180986       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:21:42.890253       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:24:31.184330       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:25:46.737800       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:27:53.892370       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:30:09.186549       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:32:43.740164       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:34:30.894569       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:37:13.189427       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:41:35.742301       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:41:42.896767       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:45:21.191783       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:50:04.898485       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:51:30.745121       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:51:47.194469       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:58:18.197354       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 19:58:25.901050       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:00:08.749244       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:03:26.199365       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:04:00.903527       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:05:28.751601       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:12:13.754355       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:13:07.202024       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:13:36.905830       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:18:49.204387       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:19:42.756231       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:21:43.907987       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:24:58.207296       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:25:39.757654       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:29:18.910043       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:32:11.760230       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:34:45.209645       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:38:10.913116       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:38:38.764405       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:43:19.212593       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:44:23.766271       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:44:28.915234       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:49:05.214788       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:50:15.917084       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:51:55.768833       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:54:56.216630       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 20:58:05.919872       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:01:14.771265       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:03:52.218531       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:06:24.773476       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:07:50.922124       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:11:44.775851       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:13:00.220314       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:16:21.924617       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:19:28.222420       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:21:05.777782       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:21:42.927208       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:27:43.225091       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:30:08.929861       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:30:38.780210       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:36:02.932003       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:37:36.226844       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:40:25.782274       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:43:39.934409       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:44:26.228825       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:47:06.788139       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:50:53.936688       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:51:50.231471       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:53:07.790760       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:59:07.793037       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 21:59:14.939203       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:00:07.233455       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:04:33.796109       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:05:37.942053       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:07:24.236266       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:10:59.798161       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:15:28.945080       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:16:39.238309       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:19:54.800156       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:21:20.947329       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:22:02.240429       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:26:37.802125       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:26:44.949622       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:29:01.243258       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:33:09.803964       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:35:10.951458       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:36:27.245654       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:42:48.806335       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:43:33.247444       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:44:51.953597       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:49:09.249617       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:51:17.809017       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:52:23.955328       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 22:57:21.252062       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:01:09.811379       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:01:38.957390       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:05:03.254891       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:06:37.813958       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:11:23.959370       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:11:27.256884       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:14:46.816633       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:18:40.961547       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:19:07.259419       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:24:30.819405       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:24:57.963735       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:26:57.261952       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:30:58.966453       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:33:10.821831       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:34:41.263645       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:37:20.968455       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:42:16.824682       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:43:41.266265       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:45:00.971256       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:51:26.827694       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:52:18.973781       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0425 23:52:38.268276       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:00:16.976022       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:01:09.830279       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:01:44.271180       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:08:08.273554       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:08:08.977891       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:09:18.832856       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:14:47.276389       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:16:59.835643       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:17:03.980211       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:19:50.279344       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:25:35.982447       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:26:06.838371       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:29:41.282117       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:34:11.841597       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:34:59.985166       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:39:15.285262       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:39:36.843841       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:41:51.987790       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:45:43.288001       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:49:21.845602       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:51:50.990108       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:55:24.290241       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:56:27.847472       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 00:57:22.992134       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:00:41.292813       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:03:24.994380       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:06:20.849819       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:08:21.295389       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:08:27.997234       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:11:27.852632       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:14:47.297304       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:16:55.855247       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:18:09.999029       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:21:18.299360       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:22:48.858089       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:27:12.001325       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:29:22.301667       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:31:14.867386       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:32:40.003534       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:36:19.303301       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:38:25.870216       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:40:17.005745       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:45:00.305646       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:45:22.872688       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:49:00.008150       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:53:29.308264       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:55:20.875077       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 01:57:01.010658       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:01:19.311259       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:02:48.013365       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:04:26.877655       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:11:16.314386       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:11:36.015295       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:13:37.879471       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:17:05.317341       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:18:19.017159       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:20:01.882163       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:23:51.019532       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:26:21.320435       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:29:40.884864       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:31:58.322207       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:33:13.021876       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:34:53.888057       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:40:41.324366       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:42:02.890498       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:42:07.024515       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:46:42.327103       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:49:28.893472       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:50:15.026862       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:54:54.329429       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:56:28.028842       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 02:59:24.896317       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:03:47.332484       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:04:28.899489       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:06:26.031704       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:10:24.902480       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:11:00.335422       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:13:34.034634       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:17:01.337760       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:19:31.904822       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:21:23.036872       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:26:22.340087       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:27:56.907048       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:28:19.039449       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:33:49.342229       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:35:32.041440       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:37:08.909744       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:41:50.345293       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:43:31.043511       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:43:40.912460       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:46:54.347571       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:50:52.046206       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:51:55.915159       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:52:12.350209       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:56:56.048150       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 03:59:18.917540       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:00:12.352338       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:03:19.050840       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:07:02.919261       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:08:42.354710       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:11:12.053166       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:14:11.922840       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:16:24.357143       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:19:20.055119       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:23:19.359722       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:23:41.924984       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:28:30.057429       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:31:05.362998       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:31:45.927301       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:36:22.060220       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:37:14.365014       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:40:39.929205       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:45:55.063046       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:47:06.367111       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:47:08.932107       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:51:16.065463       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:52:11.369513       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:55:07.933632       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 04:57:40.068026       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:00:10.372356       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:02:50.936418       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:05:20.070036       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:08:10.375216       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:11:32.938745       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:13:57.072004       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:15:47.377956       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:18:32.941392       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:20:13.074961       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:24:07.379967       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:24:30.943888       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:27:53.077648       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:29:54.382674       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:31:32.946423       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:34:25.080051       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:39:05.385491       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:39:46.082159       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:40:06.948608       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:46:06.951156       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:48:28.084599       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:48:43.390072       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:52:35.953818       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:56:24.393117       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:58:14.087362       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 05:58:17.955865       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:02:59.395092       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:06:21.089988       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:07:25.958227       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:12:13.092423       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:12:13.397449       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:17:07.960835       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:19:09.399973       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:21:46.094433       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:24:33.963414       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:26:53.097324       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:28:11.402560       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:31:18.965766       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:33:24.404774       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:34:03.099804       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:38:26.967616       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:40:59.101563       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:41:47.406427       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:44:58.970137       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:49:29.103174       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:51:06.409048       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:51:38.972774       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:55:10.105635       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:58:07.975825       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 06:59:46.411648       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:01:30.107435       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:06:32.413788       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:07:20.977676       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:07:52.110127       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:14:22.415794       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:14:32.979948       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:17:39.112680       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:20:47.982068       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:22:04.418082       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:26:17.114628       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:28:40.983624       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:30:55.420252       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:31:17.116489       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:36:04.985483       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:38:40.422613       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:40:46.118169       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:43:50.988081       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:47:34.424244       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:47:51.120709       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:50:41.990546       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:53:59.426276       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:54:34.122872       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 07:56:17.993136       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:01:39.428091       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:02:21.125094       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:04:17.995235       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:08:03.127845       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:08:26.434058       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:10:23.997129       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:14:25.436541       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:16:07.998800       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:17:22.129835       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:21:07.439819       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:22:36.001253       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:27:08.133338       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:29:42.443478       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:32:14.004215       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:35:38.445429       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:36:13.135930       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:40:47.006231       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:42:10.447232       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:46:03.138894       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:50:39.008107       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:51:08.141613       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:51:10.450051       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:56:23.010436       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 08:56:53.144645       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:00:31.452788       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:01:55.147093       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:04:20.013204       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:07:55.454734       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:08:29.149253       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:11:58.015133       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:13:24.456571       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:15:53.152003       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:17:56.017729       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:19:16.458709       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:21:07.154965       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:24:45.019795       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:27:15.461311       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:30:25.157631       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:31:20.021961       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:35:43.463017       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:36:36.159943       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:38:31.024834       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:44:34.465114       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:45:35.027210       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:46:01.162287       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:51:21.164967       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:53:16.467314       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:53:57.028885       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 09:59:15.469690       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:00:01.167376       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:02:25.031781       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:04:40.471452       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:06:10.169600       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:08:19.034537       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:11:28.473104       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:11:46.171580       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:17:18.036821       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:19:14.173483       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:21:21.475224       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:24:08.039738       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:25:10.176815       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:26:53.477127       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:30:37.179810       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:32:04.479741       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:33:08.042754       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:37:15.482197       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:38:15.182489       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:39:50.045121       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:44:35.185506       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:46:16.484789       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:48:30.047188       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:50:08.187462       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:52:34.486845       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:55:12.049242       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 10:57:26.189496       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:01:00.489444       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:02:33.051633       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:02:56.192148       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:07:55.491677       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:09:16.053762       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:10:26.194170       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:13:32.493617       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:15:56.056394       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:16:08.196117       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:20:41.495989       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:22:04.198583       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:24:25.058970       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:28:39.498591       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:29:47.200720       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:32:27.061215       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:35:58.501127       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:37:25.202941       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:41:18.503174       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:41:30.063334       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:42:26.205249       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:48:09.065648       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:48:26.505434       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:49:59.207106       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:53:54.068338       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:55:54.209664       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:56:02.507080       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 11:59:42.070674       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:03:21.212108       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:03:55.508379       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:08:00.073505       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:11:54.214690       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:13:35.510710       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:17:19.076382       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:19:51.513678       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:20:56.217112       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:24:20.078597       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:26:53.219245       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:27:11.516205       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:29:20.080955       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:34:48.221357       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:36:18.083558       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:36:31.518976       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:41:12.223953       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:41:58.085947       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:42:59.521167       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:47:24.087722       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:49:53.523614       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:51:09.226706       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:53:00.090617       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:56:09.525157       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 12:59:28.092958       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:00:37.228665       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:02:36.527584       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:07:59.094821       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:08:36.530172       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:09:45.231620       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:13:43.532786       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:15:36.097055       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:17:52.235513       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:19:20.534939       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:24:17.238395       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:24:57.099604       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:27:22.536984       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:32:05.101815       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:33:30.240833       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:37:16.539350       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:37:17.104219       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:40:38.243986       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:43:54.541764       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:44:47.106517       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:49:16.246349       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:53:12.108771       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:53:44.543504       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 13:56:18.249425       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:02:12.252120       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:02:56.545746       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:03:04.110535       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:08:44.112756       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:09:07.255405       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:11:12.548058       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:16:09.257483       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:17:53.114737       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:21:10.550239       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:25:33.116960       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:25:59.259514       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:31:01.552362       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:35:07.119515       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:35:50.262450       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:37:36.554980       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:40:55.266080       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:43:24.556937       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:43:29.121353       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:49:53.268265       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:51:50.558930       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:53:13.123477       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:55:22.271426       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 14:57:19.561228       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:02:17.126453       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:04:21.563395       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:04:56.274213       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:10:46.276155       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:11:08.129308       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:12:08.565562       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:17:29.568138       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:19:38.279261       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:20:35.131880       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:26:07.570456       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:26:11.134260       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:28:54.281246       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:34:06.139006       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:34:41.573149       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:37:53.283296       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:40:27.141536       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:43:03.576017       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:44:49.285772       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:48:28.143950       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:50:10.288254       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:53:00.578183       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:55:18.146255       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 15:58:56.290228       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:01:14.581248       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:03:15.149492       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:06:24.293087       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:09:21.583576       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:12:23.152271       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:14:50.295413       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:17:52.586327       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:21:20.298155       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:21:21.154955       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:26:27.588649       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:28:36.157012       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:29:34.300859       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:32:02.591244       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:38:21.159977       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:38:52.303602       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:41:04.593795       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:44:09.162309       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:47:36.597139       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:47:59.305633       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:50:18.164556       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:53:21.599717       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:57:30.308445       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 16:57:40.167406       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:00:26.602606       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:02:52.310620       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:07:01.605349       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:07:30.169560       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:08:19.313401       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:15:09.172324       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:15:57.608302       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:18:17.315040       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:23:58.174692       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:25:01.610589       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:26:19.317864       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:29:57.176430       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:33:55.320223       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:34:35.613043       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:35:53.179035       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:40:59.323191       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:42:09.180841       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:43:21.615425       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:46:46.325740       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:49:56.618012       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:51:53.182843       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:52:03.329002       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:56:53.185754       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:57:15.331589       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 17:58:53.619954       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:04:08.622318       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:05:21.333800       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:05:46.188586       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:09:12.624568       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:11:39.190517       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:14:56.336193       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:18:04.627067       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:19:28.192450       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:22:41.338782       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:25:41.195021       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:27:59.629816       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:31:35.341140       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:33:35.632222       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:34:45.197439       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:38:44.343302       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:41:39.635077       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:44:21.199825       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:47:41.637641       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:48:00.345838       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:51:51.202297       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:54:32.348672       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 18:55:29.640098       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:01:05.643356       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:01:37.205046       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:02:10.353859       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:09:09.646281       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:10:18.208080       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:10:22.356255       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:14:28.648163       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:17:00.357754       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:17:15.211088       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:20:04.650542       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:25:11.213044       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:25:14.359958       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:27:22.652940       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:31:59.362840       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:33:13.215847       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:36:47.655386       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:40:27.365850       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:40:32.218102       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:43:53.657484       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:47:47.368026       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:49:30.219782       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:49:55.660172       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:55:36.664211       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:56:23.222256       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 19:56:44.370790       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:01:51.666875       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:04:56.372968       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:05:25.231564       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:11:08.669118       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:11:26.234368       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:13:03.375737       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:18:33.237066       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:20:38.671502       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:21:20.378453       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:24:11.239046       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:26:46.381422       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:26:54.674267       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:30:35.242222       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:33:06.676611       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:33:09.383502       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:36:54.244998       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:39:56.385058       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:40:13.678721       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:46:15.387401       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:46:16.247193       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:48:56.681170       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:51:20.249829       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:53:04.389819       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 20:58:42.683303       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:00:27.392010       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:00:34.252721       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:05:04.685924       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:06:50.254578       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:09:20.394206       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:10:15.687843       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:12:08.256977       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:14:22.396795       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:16:48.689457       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:18:46.259504       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:22:49.399375       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:25:39.691456       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:26:48.262176       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:28:02.402241       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:34:07.693995       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:34:30.265152       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:36:22.404115       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:40:30.696074       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:42:18.267744       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:44:15.406563       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:48:36.697866       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:51:22.270433       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:52:46.408212       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:53:40.705575       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:57:24.273517       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 21:59:06.707367       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:02:37.409717       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:05:48.711082       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:06:32.276275       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:08:24.412235       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:12:27.279205       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:12:46.714338       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:16:29.414759       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:17:47.281119       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:18:49.717031       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:22:48.283512       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:23:31.416945       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:24:46.718369       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:29:59.286389       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:30:21.418729       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:32:05.720028       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:35:59.420695       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:37:33.288977       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:38:25.722603       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:41:02.423523       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:43:00.292008       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:44:52.724986       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:48:51.425672       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:49:34.294945       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:51:18.726554       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:54:36.297524       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:58:38.728310       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 22:58:50.428558       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:02:07.300426       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:04:11.730570       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:07:27.303099       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:08:31.430978       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:11:34.732726       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:14:57.433129       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:15:43.305147       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:19:04.735490       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:21:47.435809       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:24:12.307608       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:25:07.739681       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:27:02.438195       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:30:35.310582       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:30:59.742205       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:36:16.443578       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:37:03.744866       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:37:28.313495       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:44:11.446341       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:44:13.747633       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:44:23.315500       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:51:52.318159       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:52:10.749515       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:53:59.448451       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:59:16.320255       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0426 23:59:57.451364       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:00:12.751864       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:04:51.322289       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:07:08.754229       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:08:54.453706       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:11:35.324679       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:12:14.759279       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:15:19.456424       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:19:48.327390       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:20:59.761604       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:22:14.458099       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:26:33.330135       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:27:54.763553       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:28:47.460180       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:34:19.766414       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:34:59.332594       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:37:25.462185       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:42:04.768764       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:43:00.334939       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:46:47.463979       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:50:34.336723       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:50:35.771709       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:51:57.465488       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:57:29.339565       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 00:58:53.773480       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:01:10.467620       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:04:52.776553       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:05:04.342080       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:09:58.469422       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:12:17.344307       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:13:14.778714       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:15:51.471417       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:18:37.781381       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:19:25.346254       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:21:30.473460       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:26:42.784132       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:28:24.349059       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:30:59.475639       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:32:41.787481       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:34:21.352152       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:36:38.477771       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:41:58.355079       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:42:12.790150       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:42:59.480362       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:48:23.357184       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:50:46.482045       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:51:37.793264       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:54:27.360040       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 01:59:28.797880       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:00:04.484692       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:02:26.362976       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:04:39.800247       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:07:43.486802       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:09:28.365424       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:11:03.802647       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:14:14.488425       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:17:51.368227       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:17:51.804825       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:20:59.490038       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:23:20.371445       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:24:38.806183       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:26:49.492077       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:29:29.373865       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:32:14.808488       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:34:51.494023       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:36:54.376724       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:41:31.811234       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:43:08.378996       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:44:31.495538       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:50:00.381349       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:51:15.813293       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:51:42.497544       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:55:13.383683       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 02:59:53.820520       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:01:21.499419       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:03:31.386466       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:07:39.822829       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:09:17.501825       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:10:11.388344       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:15:20.825457       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:16:16.503825       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:18:44.390080       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:25:15.828345       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:25:39.506300       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:27:14.391769       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:32:11.830343       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:32:25.508279       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:32:55.393963       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:38:26.395995       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:38:58.832993       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:40:19.510589       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:44:09.397737       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:44:50.839469       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:45:34.512757       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:49:50.399600       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:52:30.841406       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:52:42.515463       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:55:15.402084       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 03:58:30.843555       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:00:23.403996       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:01:41.517651       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:04:16.846221       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:06:49.406421       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:10:06.849269       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:10:32.519873       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:13:15.408373       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:16:52.522931       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:17:02.851341       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:22:12.854139       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:23:14.410251       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:23:26.525115       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:29:53.527592       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:31:08.413149       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:31:30.856693       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:39:52.530512       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:40:57.415374       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:41:24.861301       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:45:13.532572       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:47:52.864168       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:49:51.418076       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:51:38.535327       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:55:41.420157       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:57:47.866711       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 04:58:01.537655       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:02:26.422503       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:03:31.868720       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:05:25.540122       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:08:41.425559       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:10:49.870612       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:14:00.428286       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:14:13.542072       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:19:21.872731       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:20:36.544320       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:23:16.431028       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:26:48.875589       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:28:41.546275       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:32:26.433878       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:34:15.877786       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:35:26.549197       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:39:24.880447       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:39:50.436446       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:43:42.551387       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:46:06.883299       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:47:49.439174       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:52:44.885127       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:53:29.553749       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 05:56:07.444176       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:02:14.446423       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:02:36.557279       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:02:39.888115       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:07:39.448269       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:07:55.559281       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:08:28.891538       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:15:09.450501       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:15:28.561598       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:15:55.893306       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:22:44.452597       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:23:39.563167       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:24:11.895246       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:29:29.455175       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:30:13.565268       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:33:47.896868       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:35:26.567343       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:37:37.457356       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:39:43.899249       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:43:44.460007       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:45:00.569858       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:46:12.901289       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:48:45.462012       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:51:17.903295       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:52:14.572414       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:54:35.463707       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:57:58.575116       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 06:58:05.907383       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:04:33.465559       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:05:26.577255       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:05:42.909206       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:11:10.579512       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:12:40.467141       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:15:26.911287       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:16:15.582406       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:20:47.914282       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:21:04.469814       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:22:54.584890       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:26:15.916763       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:30:15.471853       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:31:41.587644       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:32:17.918450       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:36:28.474344       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:37:31.589862       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:41:47.920217       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:45:23.476760       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:46:11.592239       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:47:08.922703       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:52:22.595121       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:55:15.478328       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 07:55:39.924847       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:01:17.480329       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:01:17.926385       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:01:28.597451       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:06:53.482135       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:10:05.600168       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:11:14.928492       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:14:35.484563       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:15:34.602402       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:20:48.930420       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:21:18.487051       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:24:13.604964       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:27:55.488544       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:29:41.932362       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:30:39.607499       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:37:13.609744       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:37:30.490926       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:38:15.935389       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:44:11.939071       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:46:17.612959       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:46:25.493443       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:49:24.941357       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:52:24.495936       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:54:09.615562       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 08:55:30.943449       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:00:27.617699       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:01:07.498699       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:02:24.945736       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:07:01.501034       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:07:15.620094       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:09:10.949665       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:13:11.622887       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:16:00.502895       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:16:37.951673       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:20:36.625373       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:21:25.505236       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:22:01.954698       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:27:42.507189       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:29:24.628096       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:29:40.957695       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:33:55.509371       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:36:00.959502       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:38:22.630129       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:41:57.962215       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:42:30.512087       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:44:01.632498       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:47:20.964756       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:50:13.514748       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:51:53.635086       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:52:32.967111       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:57:21.637626       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:58:52.969163       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 09:59:13.517132       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:03:47.640340       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:04:48.519235       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:06:10.971991       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:09:27.643200       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:10:31.521996       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:14:22.974031       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:14:45.646165       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:20:18.524951       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:20:42.648073       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:24:15.976032       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:28:08.527292       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:28:19.650996       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:32:22.978172       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:35:13.653832       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:36:53.529395       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:41:14.656121       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:41:32.980082       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:44:39.531779       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:47:19.658392       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:50:16.982289       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:53:13.534479       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:54:36.661516       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:58:29.536173       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 10:59:23.984933       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:02:01.665373       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:06:52.538595       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:07:47.987064       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:10:45.668178       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:13:07.540599       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:14:20.989075       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:17:07.670994       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:18:39.542595       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:20:02.990995       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:22:50.673661       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:26:02.993571       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:26:34.544237       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:31:09.996375       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:32:07.675482       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:34:35.546231       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:37:13.677711       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:39:52.998812       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:41:57.548227       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:45:16.679514       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:46:04.001014       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:46:57.550599       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:54:34.002841       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:54:44.682212       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 11:55:50.553127       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:01:48.556359       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:01:57.005571       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:04:32.684376       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:10:36.559015       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:11:23.008190       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:12:25.686383       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:19:54.561322       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:20:43.011445       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:21:36.688943       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:26:39.013647       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:27:05.563370       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:30:30.690487       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:34:08.015931       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:36:01.692352       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:36:34.565408       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:39:10.017805       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:43:36.694840       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:45:28.567642       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:45:55.021072       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:49:56.696759       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:53:35.569542       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:54:39.023364       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 12:56:22.699198       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:00:41.026123       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:03:00.572460       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:04:30.701234       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:05:42.028812       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:11:02.706923       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:12:22.574262       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:14:31.030821       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:16:13.709855       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:17:53.577130       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:21:21.033233       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:24:36.712883       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:27:44.580026       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:27:55.035995       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:30:48.715783       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:35:39.583242       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:37:51.038456       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:39:33.718717       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:44:08.586014       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:45:26.041042       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:47:18.721236       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:51:14.587929       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:52:51.723369       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:52:59.042759       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:58:42.589970       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 13:59:22.044598       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:02:40.726012       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:04:29.046605       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:06:20.592069       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:11:34.728438       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:12:03.049804       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:12:21.594097       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:16:58.731252       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:17:47.596501       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:18:41.052491       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:23:44.598789       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:24:01.734160       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:27:34.054833       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:29:24.601010       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:33:10.736411       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:37:14.057961       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:38:25.603569       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:40:27.738648       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:45:25.605766       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:46:09.060370       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:48:08.741282       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:52:12.607545       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:53:17.744211       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:54:05.062858       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 14:59:30.609442       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:00:45.746867       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:03:18.065897       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:06:56.748799       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:07:51.611669       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:12:10.751228       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:12:31.068861       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:13:56.615195       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:18:43.071543       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:20:11.753296       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:20:34.617284       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:25:13.755815       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:26:10.620129       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:27:08.073355       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:30:24.758756       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:31:54.622472       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:35:51.760543       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:36:04.075739       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:38:53.624558       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:42:17.077683       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:44:42.763374       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:46:47.627864       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:49:52.080590       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:52:18.765177       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:53:42.630855       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:57:58.767747       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:58:56.082732       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 15:59:07.633154       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:03:57.085043       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:07:36.770116       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:07:46.635212       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:12:58.637237       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:13:07.087185       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:16:54.772355       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:20:20.089233       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:20:22.639253       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:23:21.774848       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:29:16.091956       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:29:24.640998       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:30:32.777608       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:36:11.643053       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:37:25.094933       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:38:26.779616       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:43:49.783382       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:45:34.645397       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:47:16.097108       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:53:05.647671       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:53:25.785557       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:54:34.099379       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:59:08.787470       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 16:59:48.101364       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:00:23.649847       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:06:33.789693       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:07:25.104167       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:07:38.653038       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:13:19.106721       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:13:44.655295       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:14:36.791936       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:22:00.109536       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:22:48.657825       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:24:10.793801       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:28:19.112144       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:30:47.660204       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:33:34.796248       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:36:15.115412       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:36:19.663213       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:42:28.118209       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:43:25.799077       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:45:53.665592       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:48:32.120813       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:51:13.801685       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:55:13.667979       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:56:36.804178       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 17:58:28.123163       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:02:18.670577       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:04:03.806654       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:06:50.125583       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:07:37.673173       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:12:12.127827       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:12:50.811427       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:17:23.675702       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:18:18.814789       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:21:43.130122       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:24:45.817312       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:27:13.677792       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:29:38.133199       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:30:47.820203       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:35:33.679997       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:37:36.135222       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:39:39.822952       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:40:39.683219       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:43:11.138079       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:46:47.685622       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:48:47.140957       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:48:48.824880       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:53:05.687298       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:56:14.827489       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:56:17.143856       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 18:59:44.689445       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:04:24.145344       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:04:59.829999       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:07:31.691430       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:10:03.831980       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:13:39.148346       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:14:45.692875       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:18:23.834099       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:21:11.151322       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:22:43.695232       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:26:44.836100       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:29:33.153761       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:31:37.697568       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:34:51.838506       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:37:17.156518       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:39:01.700629       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:43:55.158532       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:44:14.840964       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:47:43.703081       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:51:21.161163       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:53:19.706640       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:53:41.843481       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:59:32.709435       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 19:59:45.163045       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:00:34.846083       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:05:14.165955       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:06:08.849034       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:09:09.711471       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:11:54.850934       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:12:48.168668       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:15:17.714176       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:20:23.171011       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:20:33.852801       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:22:05.716292       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:26:00.173181       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:28:09.855439       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:30:29.720109       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:33:52.175379       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:34:17.858444       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:38:40.722916       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:41:16.177613       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:42:45.861264       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:45:53.725505       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:47:12.180026       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:48:35.864142       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:52:17.730282       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:53:56.866270       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:56:01.182534       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 20:59:34.869234       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:01:48.732163       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:03:44.184657       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:07:03.734610       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:09:31.871841       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:10:03.186765       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:14:19.737543       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:16:50.873669       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:18:20.189195       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:23:12.739656       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:24:58.191413       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:26:10.876471       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:30:13.745264       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:31:33.194091       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:35:25.748857       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:35:42.879274       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:36:44.196464       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:42:10.751885       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:42:30.198519       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:45:32.881020       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:48:40.755161       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:50:39.201249       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:53:30.883408       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:58:31.758129       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 21:58:37.203365       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:02:51.886196       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:06:35.760377       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:07:55.204937       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:12:31.889405       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:14:26.762498       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:17:20.207222       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:20:00.891500       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:21:57.764526       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:23:57.209973       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:27:02.767205       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:29:41.893892       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:32:31.213026       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:33:02.769109       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:37:40.896439       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:40:22.215744       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:42:05.771542       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:45:30.217823       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:47:03.899300       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:48:19.773334       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:53:50.776281       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:53:55.220081       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 22:56:09.901149       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:00:17.778509       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:02:02.903149       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:03:19.222582       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:07:14.906163       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:08:17.780683       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:08:37.224787       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:15:54.908295       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:17:03.782964       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:17:41.227094       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:23:34.230319       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:23:56.785059       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:25:23.910336       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:32:53.914100       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:33:20.787557       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:33:29.232861       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:40:15.790417       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:40:26.916243       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:42:04.234867       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:46:54.919028       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:48:22.794279       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:50:54.237327       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:54:56.796731       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:55:41.921184       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0427 23:56:45.239950       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:02:31.923244       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:04:46.799275       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:06:34.242078       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:10:57.801311       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:11:12.925836       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:14:11.244517       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:19:22.803797       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:20:07.928198       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:20:23.247166       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:25:27.806495       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:28:38.930937       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:28:52.249111       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:33:51.808987       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:36:43.933665       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:37:14.251246       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:42:15.811357       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:44:40.254025       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:46:22.936117       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:48:08.813581       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:50:48.257175       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:54:31.937978       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:54:44.816361       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 00:57:38.259678       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:01:20.818890       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:03:55.261654       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:04:16.939602       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:06:31.822037       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:10:26.942304       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:13:01.265028       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:15:08.824668       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:19:53.944538       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:22:42.827436       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:22:53.267417       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:28:27.947041       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:28:39.829778       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:32:12.270156       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:34:20.833387       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:36:51.949011       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:40:33.272599       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:40:56.835818       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:45:15.951876       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:46:21.274871       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:50:18.838954       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:50:36.954630       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:54:20.278030       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 01:58:10.956821       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:00:04.841303       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:01:53.280344       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:04:17.959455       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:05:29.843169       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:11:10.964929       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:11:49.282873       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:12:54.845053       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:16:56.286016       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:18:11.967499       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:22:02.847447       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:23:17.288634       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:25:59.273832       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:25:59.281659       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:00.287837       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:00.544245       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:00.547629       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:01.291201       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:01.549683       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:02.293708       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:02.553448       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:03.298346       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:03.556773       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:04.301821       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:04.559846       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:05.305398       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:05.563353       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:06.308848       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:06.566840       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:07.312567       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:07.570247       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:08.315167       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:08.572754       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:09.317599       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:09.576623       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:10.320439       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:10.579541       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:11.323517       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:11.582413       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:12.327398       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:12.586143       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:13.330419       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:13.589630       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:14.333218       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:14.593109       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:15.258097       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:15.261929       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:15.336185       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:15.596153       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:16.264250       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:16.339195       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:16.599807       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:17.267824       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:17.342512       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:17.602319       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:18.271275       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:18.345960       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:18.605534       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:19.274132       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:19.349236       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:19.609010       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:20.276438       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:20.351892       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:20.612785       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:21.279441       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:21.354826       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:21.616590       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:22.282259       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:22.357742       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:22.620230       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:23.285191       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:23.360499       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:23.622742       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:24.289514       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:24.370577       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:24.625779       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:25.292993       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:25.374635       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:25.628526       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:26.296620       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:26.377891       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:26.632687       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:27.299965       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:27.381417       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:27.635388       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:28.304403       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:28.384980       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:28.639290       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:29.307324       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:29.387951       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:29.642611       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:30.310520       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:30.390571       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:30.646719       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:31.314203       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:31.393381       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:31.649710       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:32.316494       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:32.403934       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:32.653442       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:33.319960       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:33.407315       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:33.656488       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:34.323484       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:34.410595       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:34.660753       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:35.326107       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:35.413419       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:35.663741       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:36.329268       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:36.416405       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:36.667242       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:37.333010       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:37.419145       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:37.670794       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:38.337367       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:38.422211       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:38.674115       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:39.340696       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:39.425637       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:39.677374       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:40.343570       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:40.428325       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:40.680606       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:41.347231       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:41.430770       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:41.684378       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:42.350548       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:42.434218       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:42.687675       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:43.354264       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:43.436868       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:43.691628       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:44.357897       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:44.439887       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:44.695066       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:45.362168       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:45.446181       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:45.698631       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:46.365316       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:46.448650       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:46.702307       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:47.368689       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:47.451648       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:47.705476       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:48.372100       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:48.454432       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:48.708713       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:49.375294       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:49.457096       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:49.711842       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:50.378870       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:50.460742       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:50.714385       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:51.381867       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:51.463466       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:51.718053       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:52.384652       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:52.466590       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:52.721072       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:53.387617       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:53.468844       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:53.724107       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:54.391011       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:54.470971       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:54.726494       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:55.395361       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:55.474013       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:55.729519       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:56.398672       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:56.476308       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:56.732792       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:57.402431       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:57.479538       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:57.736358       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:58.406682       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:58.483309       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:58.739583       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:59.411134       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:59.487971       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:26:59.743087       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:00.414266       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:00.491280       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:00.745670       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:01.418223       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:01.493894       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:01.749453       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:02.421605       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:02.496343       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:02.752762       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:03.425281       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:03.499182       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:03.756526       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:04.428375       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:04.502143       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:04.758710       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:05.432201       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:05.505355       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:05.762072       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:06.435527       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:06.508599       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:06.765137       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:07.439480       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:07.511831       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:07.767793       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:08.445518       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:08.514839       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:08.771293       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:09.448415       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:09.517965       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:09.774666       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:10.451843       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:10.520666       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:10.777141       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:11.455747       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:11.523405       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:11.780993       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:12.459377       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:12.525856       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:12.783781       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:13.462057       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:13.529091       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:13.786893       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:14.465369       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:14.532173       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:14.790310       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:15.468888       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:15.535960       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:15.793641       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:16.472119       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:16.538850       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:16.796303       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:17.475810       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:17.541765       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:17.800234       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:18.479673       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:18.545776       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:18.804176       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:19.483646       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:19.548968       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:19.807863       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:20.486233       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:20.552008       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:20.810567       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:21.489221       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:21.555014       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:21.813086       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:22.492218       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:22.558597       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:22.816096       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:23.495630       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:23.561347       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:23.819841       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:24.499306       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:24.564012       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:24.823508       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:25.502536       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:25.567191       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:25.827713       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:26.506056       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:26.570260       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:26.831182       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:27.510049       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:27.572881       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:27.834689       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:28.513548       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:28.576363       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:28.838480       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:29.517610       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:29.579850       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:29.842162       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:30.521525       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:30.583379       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:30.845562       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:31.524838       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:31.587021       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:31.848528       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:32.529159       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:32.590107       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:32.852540       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:33.533088       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:33.593286       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:33.855837       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:34.536755       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:34.595656       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:34.859326       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:35.540885       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:35.599014       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:35.863200       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:35.969893       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:36.545187       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:36.602896       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:36.866041       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:37.548318       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:37.606144       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:37.869943       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:38.551392       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:38.609233       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:38.873365       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:39.555008       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:39.612828       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:39.876757       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:40.558250       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:40.615900       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:40.880037       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:41.561858       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:41.618600       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:41.883119       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:42.565596       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:42.621475       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:42.886236       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:43.569173       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:43.625018       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:43.889754       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:44.572306       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:44.627479       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:44.893139       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:45.576564       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:45.630057       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:45.896387       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:46.579669       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:46.632950       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:46.899619       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:47.583279       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:47.636005       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:47.903169       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:48.587433       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:48.639298       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:48.906807       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:49.591483       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:49.641696       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:49.910189       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:50.594893       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:50.645949       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:50.913622       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:51.599014       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:51.648851       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:51.917001       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:52.602303       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:52.651703       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:52.919730       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:53.606164       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:53.655033       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:53.922443       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:54.609647       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:54.657458       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:54.925820       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:55.613038       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:55.660662       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:55.929233       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:56.616700       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:56.663394       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:56.931928       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:57.619708       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:57.666597       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:57.935624       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:58.623186       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:58.669973       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:58.939133       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:59.626701       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:27:59Z" level=error msg="failed to save helper-pod-delete-pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916 logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 02:27:59.674523       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:27:59Z" level=info msg="clean up volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916 failed: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: create process timeout after 120 seconds" 
E0428 02:27:59.679513       1 controller.go:1558] "Volume deletion failed" err="failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: create process timeout after 120 seconds" PV="pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916"
I0428 02:27:59.679856       1 event.go:389] "Event occurred" object="pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: create process timeout after 120 seconds"
I0428 02:27:59.680159       1 controller.go:1010] "Retrying syncing volume because failures < threshold" key="pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916" failures=0 threshold=15
E0428 02:27:59.680179       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:27:59.680249       1 controller.go:1025] "Unhandled Error" err="error syncing volume \"pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916\": failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: create process timeout after 120 seconds" logger="UnhandledError"
E0428 02:27:59.942819       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:00.630255       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:28:00Z" level=error msg="failed to save helper-pod-delete-pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 02:28:00.946632       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:28:00Z" level=info msg="clean up volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c failed: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: create process timeout after 120 seconds" 
E0428 02:28:00.951034       1 controller.go:1558] "Volume deletion failed" err="failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: create process timeout after 120 seconds" PV="pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c"
I0428 02:28:00.951117       1 controller.go:1010] "Retrying syncing volume because failures < threshold" key="pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c" failures=0 threshold=15
E0428 02:28:00.951175       1 controller.go:1025] "Unhandled Error" err="error syncing volume \"pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c\": failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: create process timeout after 120 seconds" logger="UnhandledError"
I0428 02:28:00.951233       1 event.go:389] "Event occurred" object="pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: create process timeout after 120 seconds"
E0428 02:28:00.951328       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:01.633516       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:02.636815       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:03.639993       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:04.642874       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:05.646283       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:06.650290       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:07.653600       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:08.657211       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:09.660346       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:10.663548       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:11.667093       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:12.670286       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:13.673723       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:14.677040       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:14.681221       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:14.689304       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:28:15Z" level=error msg="failed to save helper-pod-delete-pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 02:28:15.680527       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:28:15Z" level=info msg="clean up volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c failed: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: create process timeout after 120 seconds" 
E0428 02:28:15.685526       1 controller.go:1558] "Volume deletion failed" err="failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: create process timeout after 120 seconds" PV="pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c"
I0428 02:28:15.685594       1 controller.go:1010] "Retrying syncing volume because failures < threshold" key="pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c" failures=0 threshold=15
E0428 02:28:15.685649       1 controller.go:1025] "Unhandled Error" err="error syncing volume \"pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c\": failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: create process timeout after 120 seconds" logger="UnhandledError"
I0428 02:28:15.685717       1 event.go:389] "Event occurred" object="pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: create process timeout after 120 seconds"
E0428 02:28:15.685869       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:15.692469       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:15.952042       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:15.956566       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:16.695031       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:16.959270       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:17.698690       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:17.962689       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:18.701542       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:18.966699       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:19.705158       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:19.969472       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:20.709094       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:20.973225       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:21.712440       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:28:21Z" level=error msg="failed to save  logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 02:28:21.714800       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:28:21Z" level=error msg="unable to delete the helper pod: pods \"helper-pod-delete-pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916\" not found" 
time="2025-04-28T02:28:21Z" level=info msg="clean up volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916 failed: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: pods \"helper-pod-delete-pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916\" not found" 
E0428 02:28:21.716773       1 controller.go:1558] "Volume deletion failed" err="failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: pods \"helper-pod-delete-pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916\" not found" PV="pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916"
I0428 02:28:21.716825       1 controller.go:1010] "Retrying syncing volume because failures < threshold" key="pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916" failures=1 threshold=15
E0428 02:28:21.716862       1 controller.go:1025] "Unhandled Error" err="error syncing volume \"pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916\": failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: pods \"helper-pod-delete-pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916\" not found" logger="UnhandledError"
I0428 02:28:21.716935       1 event.go:389] "Event occurred" object="pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: pods \"helper-pod-delete-pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916\" not found"
E0428 02:28:21.717041       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:21.976161       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:22.979504       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:23.983252       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:24.986626       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:25.989784       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:26.993020       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:27.996531       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:29.000452       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:30.003325       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:30.687033       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:30.692481       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:31.006882       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:31.695602       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:32.010393       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:32.699132       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:33.014215       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:33.702670       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:34.017897       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:34.706043       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:35.021170       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:35.709101       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:36.024555       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:36.713109       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:37.028438       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:37.717168       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:38.032327       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:38.721036       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:39.035564       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:39.724832       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:40.038995       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:40.728054       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:41.042431       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:41.731296       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:42.045964       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:42.735335       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:43.048939       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:43.739396       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:44.051825       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:44.743702       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:45.055132       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:45.747562       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:46.058944       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:46.750617       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:47.062631       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:47.754078       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:48.066525       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:48.757459       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:49.069847       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:49.761548       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:50.073849       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:50.765375       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:51.077581       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:51.717332       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:51.724995       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:51.769308       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:52.081750       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:52.727594       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:52.771506       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:53.085173       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:53.730900       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:53.774663       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:54.089029       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:54.734942       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:54.777461       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:55.093326       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:55.737659       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:55.780554       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:56.096238       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:56.741563       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:56.783295       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:57.099807       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:57.745591       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:57.786402       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:58.103127       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:58.748584       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:58.788800       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:59.106129       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:59.752659       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:28:59.792074       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:00.110167       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:00.755308       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:00.795808       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:01.113329       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:29:01Z" level=error msg="failed to save  logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 02:29:01.115683       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:29:01Z" level=error msg="unable to delete the helper pod: pods \"helper-pod-delete-pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c\" not found" 
time="2025-04-28T02:29:01Z" level=info msg="clean up volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c failed: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: pods \"helper-pod-delete-pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c\" not found" 
E0428 02:29:01.117373       1 controller.go:1558] "Volume deletion failed" err="failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: pods \"helper-pod-delete-pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c\" not found" PV="pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c"
I0428 02:29:01.117428       1 controller.go:1010] "Retrying syncing volume because failures < threshold" key="pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c" failures=1 threshold=15
E0428 02:29:01.117467       1 controller.go:1025] "Unhandled Error" err="error syncing volume \"pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c\": failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: pods \"helper-pod-delete-pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c\" not found" logger="UnhandledError"
I0428 02:29:01.117576       1 event.go:389] "Event occurred" object="pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: pods \"helper-pod-delete-pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c\" not found"
E0428 02:29:01.117732       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:01.757842       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:01.798973       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:02.761943       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:02.802160       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:03.765637       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:03.805247       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:04.769341       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:04.808125       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:05.773427       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:05.810492       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:06.776910       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:06.812776       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:07.780971       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:07.815234       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:08.785370       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:08.817843       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:09.788160       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:09.820855       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:10.791055       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:10.823259       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:11.794611       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:11.825655       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:12.798565       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:12.828471       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:13.801666       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:13.831412       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:14.804627       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:14.834544       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:15.808512       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:15.837124       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:16.811481       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:16.840306       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:17.815233       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:17.843073       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:18.819076       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:18.846172       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:19.822532       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:19.848323       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:20.825481       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:20.851002       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:21.828355       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:21.853673       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:22.831889       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:22.856601       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:23.834708       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:23.859463       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:24.838489       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:24.862349       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:25.842460       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:25.865148       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:26.845346       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:26.867721       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:27.849814       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:27.870576       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:28.853626       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:28.873365       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:29.857323       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:29.876151       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:30.861179       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:30.878672       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:31.118201       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:31.125682       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:31.864828       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:31.881375       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:32.128719       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:32.869537       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:32.884243       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:33.132677       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:33.872514       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:33.886841       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:34.136106       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:34.875602       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:34.889399       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:35.139694       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:35.879020       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:35.892190       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:29:35Z" level=error msg="failed to save  logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 02:29:35.893864       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:29:35Z" level=error msg="unable to delete the helper pod: pods \"helper-pod-delete-pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c\" not found" 
time="2025-04-28T02:29:35Z" level=info msg="clean up volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c failed: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: pods \"helper-pod-delete-pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c\" not found" 
E0428 02:29:35.895495       1 controller.go:1558] "Volume deletion failed" err="failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: pods \"helper-pod-delete-pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c\" not found" PV="pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c"
I0428 02:29:35.895553       1 controller.go:1010] "Retrying syncing volume because failures < threshold" key="pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c" failures=1 threshold=15
E0428 02:29:35.895592       1 controller.go:1025] "Unhandled Error" err="error syncing volume \"pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c\": failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: pods \"helper-pod-delete-pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c\" not found" logger="UnhandledError"
I0428 02:29:35.895637       1 event.go:389] "Event occurred" object="pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: pods \"helper-pod-delete-pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c\" not found"
E0428 02:29:35.895756       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:36.142691       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:36.881479       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:37.146482       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:37.885191       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:38.150223       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:38.888760       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:39.153738       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:39.892047       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:40.157320       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:40.895939       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:41.160497       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:41.900294       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:42.163887       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:42.904533       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:43.167802       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:43.907786       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:44.170973       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:44.910762       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:45.174989       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:45.914501       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:46.178810       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:46.918650       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:47.182409       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:47.922445       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:48.185986       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:48.926280       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:49.189199       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:49.929575       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:50.192587       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:50.932978       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:51.195473       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:51.936426       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:52.199214       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:52.940996       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:53.202436       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:53.945152       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:54.206425       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:54.948511       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:55.210197       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:55.952819       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:56.213227       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:56.956872       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:57.217023       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:57.960613       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:58.220063       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:58.964411       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:59.223949       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:29:59.969346       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:00.227715       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:00.972593       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:01.231358       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:01.977169       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:02.235088       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:02.980964       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:03.238550       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:03.985276       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:04.242554       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:04.989404       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:05.246248       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:05.896035       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:05.904846       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:05.992293       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:06.249594       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:06.907603       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:06.996077       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:07.253502       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:07.912074       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:07.999446       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:08.257216       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:08.915832       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:09.003773       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:09.261374       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:09.919748       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:10.006356       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:10.264946       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:10.923240       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:11.009284       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:11.268559       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:11.926625       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:12.012646       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:12.273058       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:12.930154       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:13.016142       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:13.276243       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:13.933314       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:14.019755       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:14.280966       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:14.937945       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:15.022959       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:15.284744       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:15.941172       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:16.025528       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:16.288713       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:16.944571       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:17.028761       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:17.292484       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:17.948850       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:18.032289       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:18.295392       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:18.952324       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:19.036027       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:19.299293       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:19.956321       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:20.039197       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:20.302717       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:20.960218       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:21.042295       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:21.305836       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:21.964297       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:22.046085       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:22.309358       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:22.967263       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:23.049425       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:23.312628       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:23.971336       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:24.053414       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:24.316639       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:24.975025       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:25.056783       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:25.320016       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:25.979388       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:26.060253       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:26.323205       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:26.983812       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:27.063762       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:27.326351       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:27.986769       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:28.066993       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:28.330121       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:28.990711       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:29.070639       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:29.333442       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:29.993791       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:30.073934       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:30.336703       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:30.997306       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:31.077201       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:31.340815       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:32.000974       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:32.080035       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:32.344161       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:33.004177       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:33.083146       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:33.347157       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:34.008060       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:34.087181       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:34.290961       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:34.351328       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:35.011369       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:35.091226       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:35.354512       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:36.014731       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:36.093987       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:36.357823       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:37.018581       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:37.096606       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:37.361829       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:38.022004       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:38.099663       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:38.365129       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:39.026558       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:39.103474       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:39.368991       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:40.029857       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:40.106099       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:40.372952       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:41.033356       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:41.110322       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:41.377145       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:42.036449       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:42.113772       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:42.380349       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:43.040699       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:43.117494       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:43.383724       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:44.045141       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:44.121299       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:44.387120       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:45.048640       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:45.124742       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:45.390438       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:46.052690       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:46.127719       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:46.394474       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:47.057273       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:47.130547       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:47.398300       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:48.060896       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:48.133525       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:48.401398       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:49.063700       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:49.136983       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:49.405165       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:50.066939       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:50.140336       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:50.409275       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:51.071014       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:51.143963       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:51.413590       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:52.076326       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:30:52Z" level=error msg="failed to save helper-pod-delete-pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916 logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 02:30:52.148192       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:30:52Z" level=info msg="clean up volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916 failed: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: create process timeout after 120 seconds" 
E0428 02:30:52.153238       1 controller.go:1558] "Volume deletion failed" err="failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: create process timeout after 120 seconds" PV="pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916"
I0428 02:30:52.153298       1 controller.go:1010] "Retrying syncing volume because failures < threshold" key="pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916" failures=2 threshold=15
E0428 02:30:52.153343       1 controller.go:1025] "Unhandled Error" err="error syncing volume \"pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916\": failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: create process timeout after 120 seconds" logger="UnhandledError"
I0428 02:30:52.153509       1 event.go:389] "Event occurred" object="pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: create process timeout after 120 seconds"
E0428 02:30:52.153594       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:52.153774       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:52.163495       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:52.416715       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:53.081196       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:53.165637       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:53.419798       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:54.084759       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:54.168670       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:54.423751       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:55.089299       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:55.172453       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:55.427185       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:56.092582       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:56.175789       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:56.430703       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:57.096028       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:57.179135       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:57.435987       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:58.099257       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:58.182627       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:58.439570       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:59.103184       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:59.185791       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:30:59.445356       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:00.106860       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:00.189272       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:00.448346       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:01.110899       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:01.192266       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:01.452131       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:02.114996       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:02.195505       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:02.454899       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:03.118558       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:03.199019       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:03.458104       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:04.124612       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:04.202878       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:04.461851       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:05.127978       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:05.206152       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:05.465400       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:06.132319       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:06.211028       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:06.471775       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:07.136246       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:07.215566       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:07.475544       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:08.139896       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:08.218637       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:08.478824       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:09.144755       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:09.222064       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:09.482974       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:10.148285       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:10.225397       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:10.485792       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:11.151763       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:11.228334       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:11.488993       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:12.155380       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:12.232495       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:12.492842       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:13.158817       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:13.236536       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:13.496204       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:14.161618       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:14.238809       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:14.499594       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:15.165593       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:15.242012       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:15.502525       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:16.177744       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:16.246348       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:16.506277       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:17.181829       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:17.249059       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:17.509345       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:18.185343       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:18.251228       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:18.512993       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:19.188689       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:19.254703       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:19.516383       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:20.195418       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:20.257665       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:20.519822       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:21.199398       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:21.261600       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:21.523662       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:22.203132       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:22.265941       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:22.527083       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:23.207308       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:23.268604       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:23.530262       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:24.211569       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:24.271362       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:24.533669       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:25.217009       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:25.275134       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:25.538257       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:26.220952       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:26.278364       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:26.541877       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:27.225449       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:27.280839       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:27.545680       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:28.229188       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:28.283840       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:28.548697       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:29.233233       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:29.287469       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:29.551837       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:30.254013       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:30.304440       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:30.555420       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:31.257384       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:31.307831       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:31:31Z" level=error msg="failed to save helper-pod-delete-pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 02:31:31.558328       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:31:31Z" level=info msg="clean up volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c failed: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: create process timeout after 120 seconds" 
E0428 02:31:31.564137       1 controller.go:1558] "Volume deletion failed" err="failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: create process timeout after 120 seconds" PV="pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c"
I0428 02:31:31.564196       1 controller.go:1010] "Retrying syncing volume because failures < threshold" key="pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c" failures=2 threshold=15
E0428 02:31:31.564256       1 controller.go:1025] "Unhandled Error" err="error syncing volume \"pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c\": failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: create process timeout after 120 seconds" logger="UnhandledError"
I0428 02:31:31.564353       1 event.go:389] "Event occurred" object="pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: create process timeout after 120 seconds"
E0428 02:31:31.564497       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:31.564547       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:31.572483       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:32.262317       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:32.310990       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:32.575186       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:33.265313       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:33.314739       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:33.578784       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:34.270931       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:34.318998       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:34.582579       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:35.274290       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:35.321741       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:35.585173       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:36.278222       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:36.324543       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:36.591547       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:37.282173       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:37.328266       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:37.595049       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:38.285288       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:38.331500       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:38.598236       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:39.288698       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:39.335010       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:39.601417       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:40.292575       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:40.337463       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:40.604829       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:41.295674       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:41.340589       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:41.608637       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:42.298769       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:42.343630       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:42.611798       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:43.302333       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:43.347206       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:43.614862       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:44.306262       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:44.349659       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:44.617684       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:45.311084       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:45.353436       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:45.621262       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:46.315197       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:46.356980       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:46.634487       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:47.319729       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:47.359944       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:47.637770       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:48.324014       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:48.363336       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:48.640965       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:49.327929       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:49.366024       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:49.644493       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:49.849934       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:50.332025       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:50.369346       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:50.648137       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:51.335672       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:51.371741       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:51.651977       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:52.339274       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:52.375439       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:52.654645       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:53.342842       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:53.378734       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:53.657863       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:54.345852       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:54.381985       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:54.661824       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:55.349873       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:55.384765       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:55.664766       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:56.353675       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:56.387531       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:56.668063       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:57.357646       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:57.390576       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:57.671371       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:58.361018       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:58.393942       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:58.675517       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:59.365158       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:59.397048       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:31:59Z" level=error msg="failed to save  logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 02:31:59.399081       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:31:59Z" level=error msg="unable to delete the helper pod: pods \"helper-pod-delete-pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916\" not found" 
time="2025-04-28T02:31:59Z" level=info msg="clean up volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916 failed: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: pods \"helper-pod-delete-pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916\" not found" 
E0428 02:31:59.400896       1 controller.go:1558] "Volume deletion failed" err="failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: pods \"helper-pod-delete-pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916\" not found" PV="pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916"
I0428 02:31:59.400970       1 controller.go:1010] "Retrying syncing volume because failures < threshold" key="pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916" failures=3 threshold=15
E0428 02:31:59.401010       1 controller.go:1025] "Unhandled Error" err="error syncing volume \"pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916\": failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: pods \"helper-pod-delete-pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916\" not found" logger="UnhandledError"
I0428 02:31:59.401104       1 event.go:389] "Event occurred" object="pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: pods \"helper-pod-delete-pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916\" not found"
E0428 02:31:59.401234       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:59.401297       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:59.405649       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:31:59.679361       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:00.368617       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:00.408163       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:00.682604       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:01.371958       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:01.411196       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:01.685324       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:02.375513       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:02.414091       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:02.689052       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:03.379255       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:03.417114       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:03.692626       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:04.382780       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:04.419768       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:04.696248       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:05.385956       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:05.422890       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:05.699262       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:32:06Z" level=error msg="failed to save helper-pod-delete-pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 02:32:06.389662       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:32:06Z" level=info msg="clean up volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c failed: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: create process timeout after 120 seconds" 
E0428 02:32:06.394546       1 controller.go:1558] "Volume deletion failed" err="failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: create process timeout after 120 seconds" PV="pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c"
I0428 02:32:06.394625       1 controller.go:1010] "Retrying syncing volume because failures < threshold" key="pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c" failures=2 threshold=15
E0428 02:32:06.394688       1 controller.go:1025] "Unhandled Error" err="error syncing volume \"pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c\": failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: create process timeout after 120 seconds" logger="UnhandledError"
I0428 02:32:06.394784       1 event.go:389] "Event occurred" object="pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: create process timeout after 120 seconds"
E0428 02:32:06.394932       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:06.394990       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:06.403220       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:06.425215       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:06.701706       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:07.405953       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:07.428116       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:07.704602       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:08.409859       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:08.430744       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:08.708300       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:09.413033       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:09.433707       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:09.712017       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:10.416216       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:10.436455       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:10.715090       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:11.420059       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:11.439082       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:11.717554       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:12.423651       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:12.441608       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:12.720965       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:13.427163       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:13.444221       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:13.724349       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:14.431349       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:14.446357       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:14.726946       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:15.435236       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:15.448745       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:15.729857       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:16.439551       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:16.452338       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:16.733218       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:17.446391       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:17.455071       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:17.736189       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:18.449320       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:18.458056       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:18.739734       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:19.452596       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:19.460611       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:19.743163       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:20.456037       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:20.462813       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:20.746523       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:21.459896       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:21.465775       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:21.749333       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:22.463098       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:22.468643       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:22.753138       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:23.466340       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:23.471251       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:23.756544       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:24.469805       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:24.473542       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:24.760301       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:25.473480       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:25.476118       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:25.763692       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:26.476859       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:26.478991       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:26.766479       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:27.480024       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:27.482059       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:27.769389       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:28.483039       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:28.485050       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:28.772618       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:29.486212       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:29.488188       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:29.776653       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:30.490354       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:30.490549       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:30.780119       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:31.494469       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:31.494631       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:31.783191       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:32.498535       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:32.498712       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:32.786231       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:33.502727       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:33.502929       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:33.789841       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:34.506750       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:34.506946       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:34.793292       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:35.510513       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:35.510697       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:35.796826       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:36.514034       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:36.514239       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:36.800224       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:37.517090       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:37.517302       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:37.803369       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:32:37Z" level=error msg="failed to save  logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 02:32:37.805666       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:32:37Z" level=error msg="unable to delete the helper pod: pods \"helper-pod-delete-pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c\" not found" 
time="2025-04-28T02:32:37Z" level=info msg="clean up volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c failed: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: pods \"helper-pod-delete-pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c\" not found" 
E0428 02:32:37.807412       1 controller.go:1558] "Volume deletion failed" err="failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: pods \"helper-pod-delete-pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c\" not found" PV="pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c"
I0428 02:32:37.807466       1 controller.go:1010] "Retrying syncing volume because failures < threshold" key="pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c" failures=3 threshold=15
E0428 02:32:37.807506       1 controller.go:1025] "Unhandled Error" err="error syncing volume \"pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c\": failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: pods \"helper-pod-delete-pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c\" not found" logger="UnhandledError"
I0428 02:32:37.807568       1 event.go:389] "Event occurred" object="pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: pods \"helper-pod-delete-pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c\" not found"
E0428 02:32:37.807728       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:37.807828       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:37.814591       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:38.520362       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:38.520591       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:38.817466       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:39.524059       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:39.524216       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:39.821025       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:40.526830       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:40.527026       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:40.824046       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:41.530023       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:41.530194       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:41.827078       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:42.533090       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:42.533327       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:42.829501       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:43.536484       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:43.536685       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:43.832634       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:44.539465       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:44.539660       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:44.835455       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:45.543367       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:45.543531       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:45.838204       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:46.546656       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:46.546776       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:46.840728       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:47.549589       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:47.549743       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:47.844963       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:48.552759       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:48.552843       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:48.848789       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:49.555618       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:49.555762       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:49.852741       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:50.559320       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:50.559495       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:50.856114       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:51.562261       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:51.562431       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:51.859588       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:52.565506       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:52.565635       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:52.862070       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:53.569705       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:53.569801       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:53.865630       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:54.572516       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:54.572709       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:54.868809       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:55.576258       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:55.576376       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:55.872203       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:56.580082       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:56.580281       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:56.875468       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:57.583860       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:57.584067       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:57.878276       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:58.587615       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:58.587801       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:58.881781       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:59.591303       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:59.591512       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:32:59.884944       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:00.595209       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:00.595383       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:00.887985       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:01.598396       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:01.598684       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:01.890760       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:02.602334       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:02.602549       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:02.893883       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:03.606400       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:03.606601       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:03.897713       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:04.610114       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:04.610299       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:04.900509       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:05.613781       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:05.613962       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:05.904033       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:06.617178       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:06.617343       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:06.908059       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:07.620392       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:07.620552       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:07.910893       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:08.623945       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:08.624108       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:08.913714       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:09.628234       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:09.628428       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:09.917264       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:10.631400       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:10.631580       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:10.920511       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:11.634311       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:11.634491       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:11.924014       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:12.638081       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:12.638311       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:33:12Z" level=error msg="failed to save  logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 02:33:12.641037       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:33:12Z" level=error msg="unable to delete the helper pod: pods \"helper-pod-delete-pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c\" not found" 
time="2025-04-28T02:33:12Z" level=info msg="clean up volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c failed: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: pods \"helper-pod-delete-pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c\" not found" 
E0428 02:33:12.643071       1 controller.go:1558] "Volume deletion failed" err="failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: pods \"helper-pod-delete-pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c\" not found" PV="pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c"
I0428 02:33:12.643160       1 controller.go:1010] "Retrying syncing volume because failures < threshold" key="pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c" failures=3 threshold=15
E0428 02:33:12.643207       1 controller.go:1025] "Unhandled Error" err="error syncing volume \"pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c\": failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: pods \"helper-pod-delete-pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c\" not found" logger="UnhandledError"
I0428 02:33:12.643297       1 event.go:389] "Event occurred" object="pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: pods \"helper-pod-delete-pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c\" not found"
E0428 02:33:12.643444       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:12.643519       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:12.650259       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:12.927496       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:13.641595       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:13.652358       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:13.930823       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:14.645285       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:14.654812       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:14.934274       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:15.648341       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:15.657264       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:15.936650       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:15.971194       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:16.651355       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:16.659711       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:16.939833       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:17.654191       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:17.662348       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:17.942748       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:18.657998       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:18.665466       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:18.945787       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:19.661770       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:19.667782       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:19.949850       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:20.672128       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:20.672303       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:20.952478       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:21.675691       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:21.675848       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:21.955691       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:22.678946       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:22.679098       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:22.958637       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:23.682798       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:23.682992       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:23.961813       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:24.686103       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:24.686311       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:24.965038       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:25.690267       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:25.690485       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:25.969187       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:26.694082       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:26.694269       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:26.973017       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:27.696887       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:27.697033       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:27.976355       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:28.701339       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:28.701514       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:28.979722       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:29.705050       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:29.705260       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:29.983364       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:30.708182       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:30.708362       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:30.986172       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:31.710935       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:31.711105       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:31.989041       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:32.714932       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:32.715100       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:32.992411       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:33.718516       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:33.718706       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:33.995060       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:34.721735       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:34.721944       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:34.997948       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:35.725743       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:35.725900       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:36.001380       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:36.730108       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:36.730269       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:37.004527       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:37.733322       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:37.733494       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:38.007816       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:38.736843       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:38.737077       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:39.011434       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:39.740417       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:39.740580       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:40.014762       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:40.744325       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:40.744506       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:41.018399       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:41.747376       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:41.747535       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:42.020940       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:42.750895       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:42.751081       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:43.024499       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:43.754896       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:43.755078       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:44.027406       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:44.758642       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:44.758798       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:45.030608       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:45.762093       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:45.762284       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:46.033482       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:46.765409       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:46.765581       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:47.037339       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:47.769309       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:47.769494       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:48.040694       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:48.773161       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:48.773322       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:49.043612       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:49.776866       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:49.777071       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:50.047784       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:50.781385       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:50.781484       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:51.051056       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:51.785569       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:51.785725       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:52.054706       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:52.788658       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:52.788812       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:53.058180       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:53.792319       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:53.792485       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:54.060874       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:54.795544       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:54.795707       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:55.064894       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:55.799444       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:55.799625       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:56.068551       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:56.803333       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:56.803493       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:57.072607       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:57.807256       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:57.807423       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:58.075837       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:58.811188       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:58.811378       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:59.079466       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:33:59Z" level=error msg="failed to save helper-pod-delete-pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916 logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 02:33:59.814408       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:59.814484       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:33:59Z" level=info msg="clean up volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916 failed: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: create process timeout after 120 seconds" 
E0428 02:33:59.818531       1 controller.go:1558] "Volume deletion failed" err="failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: create process timeout after 120 seconds" PV="pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916"
I0428 02:33:59.818592       1 controller.go:1010] "Retrying syncing volume because failures < threshold" key="pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916" failures=4 threshold=15
E0428 02:33:59.818639       1 controller.go:1025] "Unhandled Error" err="error syncing volume \"pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916\": failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: create process timeout after 120 seconds" logger="UnhandledError"
I0428 02:33:59.818720       1 event.go:389] "Event occurred" object="pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: create process timeout after 120 seconds"
E0428 02:33:59.818876       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:59.818998       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:33:59.825419       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:00.082679       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:00.817467       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:00.828485       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:01.085831       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:01.820897       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:01.831483       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:02.088896       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:02.824017       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:02.834107       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:03.091447       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:03.827506       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:03.836403       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:04.093666       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:04.830716       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:04.839357       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:05.096995       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:05.833566       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:05.842653       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:06.100631       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:06.836835       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:06.845611       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:07.103353       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:07.839964       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:07.848098       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:08.107063       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:08.844393       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:08.851602       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:09.110613       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:09.847952       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:09.854739       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:10.114559       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:10.851945       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:10.858240       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:11.117825       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:11.855861       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:11.860876       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:12.120995       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:12.859218       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:12.864464       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:13.123622       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:13.862850       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:13.866842       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:14.126426       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:14.866218       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:14.869604       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:15.130054       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:15.869438       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:15.871887       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:16.133622       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:16.873338       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:16.874353       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:17.136717       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:17.876479       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:17.876666       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:18.140410       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:18.880083       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:18.880268       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:19.143521       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:19.884053       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:19.884245       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:20.148233       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:20.887356       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:20.887515       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:21.151753       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:21.890588       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:21.890757       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:22.155264       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:22.895023       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:22.895194       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:23.158613       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:23.899321       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:23.899487       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:24.161765       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:24.902569       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:24.902729       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:25.165848       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:25.905963       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:25.906167       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:26.171049       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:26.909258       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:26.909433       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:27.175028       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:27.912301       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:27.912463       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:28.178227       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:28.916410       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:28.916570       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:29.180886       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:29.920602       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:29.920759       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:30.184838       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:30.924660       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:30.924858       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:31.188194       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:31.928280       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:31.928447       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:32.191541       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:32.931405       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:32.931587       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:33.194302       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:33.935462       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:33.935623       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:34.196774       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:34.938355       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:34.938519       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:35.200337       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:35.941179       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:35.941384       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:36.204493       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:36.944254       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:36.944407       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:37.207775       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:37.948171       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:37.948329       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:34:38Z" level=error msg="failed to save helper-pod-delete-pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 02:34:38.211379       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:34:38Z" level=info msg="clean up volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c failed: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: create process timeout after 120 seconds" 
E0428 02:34:38.216136       1 controller.go:1558] "Volume deletion failed" err="failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: create process timeout after 120 seconds" PV="pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c"
I0428 02:34:38.216201       1 controller.go:1010] "Retrying syncing volume because failures < threshold" key="pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c" failures=4 threshold=15
E0428 02:34:38.216248       1 controller.go:1025] "Unhandled Error" err="error syncing volume \"pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c\": failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: create process timeout after 120 seconds" logger="UnhandledError"
I0428 02:34:38.216307       1 event.go:389] "Event occurred" object="pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: create process timeout after 120 seconds"
E0428 02:34:38.216480       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:38.216535       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:38.223507       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:38.951056       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:38.951240       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:39.225450       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:39.954834       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:39.954997       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:40.228431       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:40.958666       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:40.958848       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:41.231778       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:41.962111       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:41.962284       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:42.235555       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:42.965389       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:42.965568       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:43.238448       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:43.969951       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:43.970164       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:44.241380       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:44.974197       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:44.974352       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:45.245172       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:45.977640       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:45.977832       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:46.248557       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:46.981947       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:46.982184       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:47.252410       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:47.985300       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:47.985481       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:48.255670       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:48.988976       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:48.989143       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:49.258870       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:49.992972       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:49.993190       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:50.262791       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:50.997023       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:50.997215       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:51.266303       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:52.000743       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:52.000958       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:52.270023       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:53.004129       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:53.004239       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:53.272722       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:54.007881       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:54.008094       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:54.276390       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:55.011813       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:55.011970       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:55.280057       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:56.015965       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:56.016142       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:56.283623       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:57.020130       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:57.020288       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:57.288594       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:58.024394       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:58.024647       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:58.291622       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:59.027659       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:59.027858       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:34:59.294786       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:00.031514       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:00.031761       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:00.298553       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:01.034347       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:01.034542       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:01.302148       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:02.037065       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:02.037238       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:02.304982       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:03.040981       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:03.041190       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:03.309246       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:04.044692       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:04.044894       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:04.312838       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:05.048644       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:05.048804       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:05.316715       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:06.051758       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:06.051957       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:06.321187       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:07.055046       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:07.055305       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:07.324075       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:08.058075       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:08.058240       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:08.327223       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:09.061524       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:09.061681       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:09.330619       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:10.065553       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:10.065718       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:10.333505       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:11.068569       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:11.068770       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:11.337337       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:12.072478       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:12.072644       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:12.339706       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:35:13Z" level=error msg="failed to save helper-pod-delete-pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 02:35:13.075498       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:13.075658       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:35:13Z" level=info msg="clean up volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c failed: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: create process timeout after 120 seconds" 
E0428 02:35:13.080082       1 controller.go:1558] "Volume deletion failed" err="failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: create process timeout after 120 seconds" PV="pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c"
I0428 02:35:13.080150       1 controller.go:1010] "Retrying syncing volume because failures < threshold" key="pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c" failures=4 threshold=15
E0428 02:35:13.080216       1 controller.go:1025] "Unhandled Error" err="error syncing volume \"pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c\": failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: create process timeout after 120 seconds" logger="UnhandledError"
E0428 02:35:13.080516       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:13.080661       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
I0428 02:35:13.080692       1 event.go:389] "Event occurred" object="pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: create process timeout after 120 seconds"
E0428 02:35:13.087986       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:13.342937       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:14.079356       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:14.090834       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:35:14Z" level=error msg="failed to save  logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 02:35:14.092604       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:35:14Z" level=error msg="unable to delete the helper pod: pods \"helper-pod-delete-pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c\" not found" 
time="2025-04-28T02:35:14Z" level=info msg="clean up volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c failed: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: pods \"helper-pod-delete-pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c\" not found" 
E0428 02:35:14.094228       1 controller.go:1558] "Volume deletion failed" err="failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: pods \"helper-pod-delete-pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c\" not found" PV="pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c"
I0428 02:35:14.094285       1 controller.go:1010] "Retrying syncing volume because failures < threshold" key="pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c" failures=5 threshold=15
E0428 02:35:14.094323       1 controller.go:1025] "Unhandled Error" err="error syncing volume \"pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c\": failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: pods \"helper-pod-delete-pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c\" not found" logger="UnhandledError"
I0428 02:35:14.094366       1 event.go:389] "Event occurred" object="pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: pods \"helper-pod-delete-pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c\" not found"
E0428 02:35:14.094587       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:14.346817       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:15.083029       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:15.349730       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:16.086598       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:16.352446       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:17.089433       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:17.355574       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:18.092707       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:18.359085       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:19.096009       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:19.362592       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:20.100015       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:20.365891       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:21.102780       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:21.368482       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:22.105499       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:22.371328       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:23.109201       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:23.373634       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:24.112843       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:24.377213       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:25.116542       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:25.380043       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:26.120864       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:26.383129       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:27.124655       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:27.387002       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:28.128663       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:28.390217       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:29.132366       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:29.393378       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:30.135390       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:30.397324       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:31.139378       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:31.399743       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:32.142999       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:32.402998       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:33.146109       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:33.405985       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:34.149855       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:34.410269       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:35.153779       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:35.413990       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:36.157273       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:35:36Z" level=error msg="failed to save  logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 02:35:36.159534       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:35:36Z" level=error msg="unable to delete the helper pod: pods \"helper-pod-delete-pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916\" not found" 
time="2025-04-28T02:35:36Z" level=info msg="clean up volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916 failed: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: pods \"helper-pod-delete-pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916\" not found" 
E0428 02:35:36.161356       1 controller.go:1558] "Volume deletion failed" err="failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: pods \"helper-pod-delete-pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916\" not found" PV="pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916"
I0428 02:35:36.161416       1 controller.go:1010] "Retrying syncing volume because failures < threshold" key="pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916" failures=5 threshold=15
E0428 02:35:36.161455       1 controller.go:1025] "Unhandled Error" err="error syncing volume \"pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916\": failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: pods \"helper-pod-delete-pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916\" not found" logger="UnhandledError"
I0428 02:35:36.161558       1 event.go:389] "Event occurred" object="pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: pods \"helper-pod-delete-pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916\" not found"
E0428 02:35:36.161777       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:36.417707       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:37.421432       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:38.425214       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:39.428629       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:40.432443       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:41.435819       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:42.292364       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:42.439832       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:43.444662       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:44.447839       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:45.450641       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:46.453975       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:47.456950       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:48.459574       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:49.462816       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:50.465540       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:51.468261       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:52.471047       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:53.473860       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:54.477073       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:55.480911       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:56.485086       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:57.488642       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:58.491582       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:35:59.494459       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:36:00.497216       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:36:01.500162       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:36:02.502795       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:36:03.505593       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:36:04.509005       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:36:05.511686       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:36:06.515754       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:36:07.518446       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:36:08.521233       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:36:09.523999       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:36:10.526897       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:36:11.529759       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:36:12.532528       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:36:13.535240       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:36:14.538472       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:36:14Z" level=error msg="failed to save  logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 02:36:14.540587       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:36:14Z" level=error msg="unable to delete the helper pod: pods \"helper-pod-delete-pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c\" not found" 
time="2025-04-28T02:36:14Z" level=info msg="clean up volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c failed: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: pods \"helper-pod-delete-pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c\" not found" 
E0428 02:36:14.542310       1 controller.go:1558] "Volume deletion failed" err="failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: pods \"helper-pod-delete-pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c\" not found" PV="pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c"
I0428 02:36:14.542363       1 controller.go:1010] "Retrying syncing volume because failures < threshold" key="pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c" failures=5 threshold=15
E0428 02:36:14.542404       1 controller.go:1025] "Unhandled Error" err="error syncing volume \"pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c\": failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: pods \"helper-pod-delete-pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c\" not found" logger="UnhandledError"
I0428 02:36:14.542500       1 event.go:389] "Event occurred" object="pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: pods \"helper-pod-delete-pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c\" not found"
E0428 02:36:14.542674       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:37:59.821571       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:37:59.828888       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:00.831336       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:01.834458       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:02.838202       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:03.841340       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:04.844499       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:05.847552       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:06.850434       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:07.853158       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:08.852088       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:08.856484       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:09.859362       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:10.863011       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:11.866270       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:12.870214       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:13.873108       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:14.876313       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:15.879665       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:16.883084       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:17.886594       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:18.889748       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:19.893728       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:20.896825       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:21.900607       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:21.972724       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:22.903689       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:23.907067       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:24.910025       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:25.913101       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:26.915802       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:27.919205       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:28.922118       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:29.925210       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:30.928631       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:31.931413       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:32.935260       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:33.938236       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:34.941727       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:35.944771       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:36.948235       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:37.951164       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:38.216751       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:38.223828       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:38.954414       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:39.226585       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:39.957845       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:40.229697       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:40.961428       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:41.232986       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:41.965275       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:42.236465       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:42.969075       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:43.239130       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:43.972331       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:44.242542       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:44.975561       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:45.246760       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:45.978627       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:46.249597       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:46.981787       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:47.253032       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:47.985116       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:48.256334       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:48.989222       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:49.259332       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:49.992240       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:50.262281       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:50.995170       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:51.265234       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:51.998702       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:52.268732       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:53.002245       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:53.271252       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:54.005222       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:54.274828       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:55.008091       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:55.278018       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:56.011631       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:56.280581       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:57.014651       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:57.283841       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:58.017906       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:58.286152       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:59.021365       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:38:59.289659       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:00.025503       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:00.292788       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:01.029629       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:01.296584       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:02.032380       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:02.299246       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:03.035391       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:03.302536       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:04.038759       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:04.305942       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:05.042019       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:05.309248       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:06.045310       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:06.312548       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:07.049387       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:07.315493       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:08.052932       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:08.318426       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:09.056647       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:09.321509       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:10.059635       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:10.325014       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:11.063346       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:11.328409       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:12.067773       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:12.331120       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:13.071100       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:13.080994       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:13.087682       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:13.333615       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:14.074762       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:14.089723       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:14.337035       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:15.077898       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:15.092685       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:15.340146       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:16.081306       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:16.094948       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:16.344551       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:17.084573       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:17.097224       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:17.347558       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:18.087652       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:18.100325       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:18.350819       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:19.091093       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:19.102734       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:19.354156       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:20.094154       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:20.105943       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:20.357269       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:21.097430       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:21.108884       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:21.360383       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:22.100604       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:22.111583       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:22.363742       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:23.104525       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:23.114603       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:23.365802       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:24.107886       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:24.117009       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:24.369393       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:25.111162       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:25.119749       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:25.372090       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:26.114642       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:26.122396       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:26.375776       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:27.117629       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:27.124735       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:27.379941       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:28.121005       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:28.126955       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:28.382585       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:29.124727       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:29.129634       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:29.385168       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:30.128437       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:30.132551       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:30.387988       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:31.131845       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:31.134710       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:31.390950       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:32.135130       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:32.137213       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:32.393503       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:33.138427       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:33.139483       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:33.395579       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:34.142008       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:34.142160       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:34.398175       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:35.146300       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:35.146482       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:35.400936       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:36.150733       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:36.150884       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:36.404715       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:37.153862       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:37.154035       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:37.408033       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:38.157087       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:38.157267       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:38.411308       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:39.160358       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:39.160509       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:39.414558       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:40.163770       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:40.163952       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:40.417106       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:41.172135       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:41.172303       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:41.420011       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:42.175119       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:42.175276       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:42.422505       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:43.178599       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:43.178751       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:43.425172       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:44.182031       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:44.182189       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:44.427556       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:45.185487       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:45.186501       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:45.430005       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:46.189414       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:46.189565       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:46.432878       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:47.193194       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:47.193440       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:47.436030       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:48.197011       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:48.197166       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:48.439164       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:49.200261       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:49.200420       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:49.445389       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:50.203658       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:50.203822       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:50.448086       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:51.207027       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:51.207179       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:51.451045       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:52.210650       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:52.210811       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:52.454060       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:53.214287       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:53.214447       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:53.456765       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:54.218198       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:54.218397       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:54.459615       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:55.221965       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:55.222278       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:55.463195       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:56.225832       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:56.225896       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:56.466089       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:57.229577       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:57.229812       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:57.469884       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:58.233333       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:58.233553       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:58.473226       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:59.236560       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:59.236740       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:39:59.475801       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:40:00Z" level=error msg="failed to save helper-pod-delete-pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916 logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 02:40:00.240412       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:00.240540       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:40:00Z" level=info msg="clean up volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916 failed: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: create process timeout after 120 seconds" 
E0428 02:40:00.245191       1 controller.go:1558] "Volume deletion failed" err="failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: create process timeout after 120 seconds" PV="pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916"
I0428 02:40:00.245264       1 controller.go:1010] "Retrying syncing volume because failures < threshold" key="pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916" failures=6 threshold=15
E0428 02:40:00.245307       1 controller.go:1025] "Unhandled Error" err="error syncing volume \"pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916\": failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: create process timeout after 120 seconds" logger="UnhandledError"
I0428 02:40:00.245354       1 event.go:389] "Event occurred" object="pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: create process timeout after 120 seconds"
E0428 02:40:00.245598       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:00.478870       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:01.244249       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:01.481197       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:02.247138       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:02.484451       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:03.250657       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:03.488026       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:04.254600       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:04.491724       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:05.257545       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:05.494840       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:06.260978       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:06.498639       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:07.264705       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:07.501755       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:08.267685       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:08.504536       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:09.270711       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:09.508049       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:10.274191       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:10.511468       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:11.278754       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:11.514527       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:12.281884       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:12.518296       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:13.285603       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:13.521851       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:14.289304       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:14.525654       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:15.292810       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:15.528896       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:16.296535       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:16.532789       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:17.300675       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:17.537165       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:18.305199       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:18.540603       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:19.308632       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:19.543142       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:20.312185       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:20.546469       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:21.315464       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:21.549190       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:22.318545       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:22.552083       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:23.321833       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:23.554805       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:24.324698       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:24.558320       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:25.328520       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:25.561669       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:26.332152       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:26.564793       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:27.336031       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:27.568246       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:28.339524       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:28.571969       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:29.343295       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:29.575647       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:30.347161       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:30.578880       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:31.351179       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:31.582173       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:32.354467       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:32.585329       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:33.357601       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:33.588611       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:34.361525       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:34.592009       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:35.365176       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:35.595528       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:36.369240       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:36.599296       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:37.372502       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:37.602341       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:38.375331       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:40:38Z" level=error msg="failed to save helper-pod-delete-pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 02:40:38.605025       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:40:38Z" level=info msg="clean up volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c failed: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: create process timeout after 120 seconds" 
E0428 02:40:38.609710       1 controller.go:1558] "Volume deletion failed" err="failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: create process timeout after 120 seconds" PV="pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c"
I0428 02:40:38.609780       1 controller.go:1010] "Retrying syncing volume because failures < threshold" key="pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c" failures=6 threshold=15
E0428 02:40:38.609831       1 controller.go:1025] "Unhandled Error" err="error syncing volume \"pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c\": failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: create process timeout after 120 seconds" logger="UnhandledError"
I0428 02:40:38.609929       1 event.go:389] "Event occurred" object="pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: create process timeout after 120 seconds"
E0428 02:40:38.610264       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:39.378871       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:40.382355       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:41.386298       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:42.389389       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:43.392156       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:44.395528       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:45.398864       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:46.403005       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:47.406797       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:48.411018       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:49.414871       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:50.418028       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:51.421797       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:52.425806       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:53.429387       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:54.432643       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:55.436043       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:56.440546       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:57.446054       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:58.449354       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:40:59.452466       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:41:00.455549       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:41:01.458853       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:41:02.462083       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:41:03.465282       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:41:04.468073       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:41:05.471411       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:41:06.475547       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:41:07.478825       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:41:08.484563       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:41:09.488004       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:41:10.493464       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:41:11.497454       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:41:12.500854       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:41:13Z" level=error msg="failed to save helper-pod-delete-pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 02:41:13.503940       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:41:13Z" level=info msg="clean up volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c failed: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: create process timeout after 120 seconds" 
E0428 02:41:13.509084       1 controller.go:1558] "Volume deletion failed" err="failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: create process timeout after 120 seconds" PV="pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c"
I0428 02:41:13.509160       1 controller.go:1010] "Retrying syncing volume because failures < threshold" key="pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c" failures=6 threshold=15
E0428 02:41:13.509232       1 controller.go:1025] "Unhandled Error" err="error syncing volume \"pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c\": failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: create process timeout after 120 seconds" logger="UnhandledError"
I0428 02:41:13.509288       1 event.go:389] "Event occurred" object="pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: create process timeout after 120 seconds"
E0428 02:41:13.509528       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:00.854848       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:03.294418       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:28.975029       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:28.975161       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:28.975246       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:28.983575       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:28.985149       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:28.985328       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:29.988323       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:29.988511       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:29.988702       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:30.991573       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:30.991704       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:30.991838       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:31.995606       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:31.995704       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:31.995758       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:32.999213       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:32.999374       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:32.999440       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:34.002776       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:34.002947       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:34.003000       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:35.006414       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:35.006557       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:35.006605       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:36.010815       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:36.010996       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:36.011125       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:37.014387       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:37.014578       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:37.014720       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:38.017843       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:38.017987       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:38.018136       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:39.021398       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:39.021526       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:39.021606       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:40.025307       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:40.025521       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:40.025652       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:41.028624       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:41.028707       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:41.028774       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:42.032047       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:42.032169       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:42.032280       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:43.035630       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:43.035836       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:43.035888       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:44.039388       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:44.039639       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:44.039801       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:45.042606       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:45.042662       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:45.042728       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:46.046190       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:46.046293       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:46.046359       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:47.049795       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:47.049953       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:47.050056       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:48.053060       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:48.053244       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:48.053335       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:49.056526       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:49.056642       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:49.056730       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:50.060513       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:50.060637       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:50.060742       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:51.064049       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:51.064184       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:51.064284       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:52.067340       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:52.067541       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:52.067697       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:53.070765       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:53.070959       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:53.071011       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:54.074594       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:54.074757       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:54.074811       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:55.078035       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:55.078179       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:55.078326       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:56.082049       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:56.082223       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:56.082378       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:57.086752       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:57.086879       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:57.086962       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:58.091244       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:58.091335       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:58.091384       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:59.095729       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:59.095901       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:45:59.095987       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:00.099969       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:00.100063       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:00.100118       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:01.103792       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:01.104012       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:01.104176       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:02.107392       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:02.107486       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:02.107536       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:03.110830       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:03.110932       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:03.110979       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:03.974881       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:04.114571       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:04.114657       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:04.114697       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:05.118610       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:05.118823       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:05.118890       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:06.121816       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:06.121935       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:06.121980       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:07.124988       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:07.125237       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:07.125332       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:08.128370       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:08.128509       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:08.128654       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:09.131141       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:09.131341       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:09.131487       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:10.134811       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:10.134958       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:10.135040       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:11.139338       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:11.139533       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:11.139640       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:12.142956       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:12.143110       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:12.143150       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:13.146535       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:13.146700       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:13.146750       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:14.149691       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:14.150011       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:14.150094       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:15.152798       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:15.152885       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:15.152950       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:16.156706       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:16.156878       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:16.157035       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:17.160527       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:17.160896       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:17.160991       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:18.164790       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:18.164944       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:18.165012       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:19.168171       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:19.168313       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:19.168381       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:20.172091       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:20.172309       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:20.172440       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:21.175315       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:21.175556       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:21.175681       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:22.178867       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:22.179005       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:22.179067       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:23.182585       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:23.182688       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:23.182748       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:24.186173       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:24.186431       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:24.186615       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:25.190109       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:25.190211       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:25.190302       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:26.193874       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:26.193982       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:26.194035       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:27.197426       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:27.197623       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:27.197938       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:28.201132       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:28.201343       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:28.201493       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:29.203888       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:29.204119       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:29.204284       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:30.208142       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:30.208405       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:30.208596       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:31.211383       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:31.211493       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:31.211568       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:32.214860       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:32.215003       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:32.215086       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:33.219235       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:33.219302       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:33.219442       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:34.222338       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:34.222454       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:34.222569       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:35.225667       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:35.225832       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:35.226007       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:36.228843       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:36.229064       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:36.229150       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:37.232320       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:37.232382       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:37.232439       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:38.235801       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:38.235974       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:38.236099       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:39.238999       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:39.239163       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:39.239582       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:40.242794       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:40.242941       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:40.243028       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:41.247224       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:41.247385       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:41.247494       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:42.250409       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:42.250594       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:42.250746       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:43.253721       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:43.253873       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:43.254042       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:44.256805       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:44.256956       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:44.257046       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:45.260335       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:45.260487       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:45.260614       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:46.264091       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:46.264175       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:46.264254       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:47.267470       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:47.267614       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:47.267796       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:48.271080       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:48.271259       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:48.271409       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:49.274527       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:49.274659       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:49.274752       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:50.278498       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:50.278686       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:50.278818       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:51.281695       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:51.281827       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:51.281887       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:52.285279       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:52.285470       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:52.285933       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:53.288506       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:53.288602       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:53.288644       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:54.292286       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:54.292386       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:54.292451       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:55.295611       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:55.295738       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:55.295812       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:56.299055       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:56.299256       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:56.299378       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:57.302001       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:57.302187       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:57.302320       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:58.304991       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:58.305090       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:58.305144       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:59.308170       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:59.308368       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:46:59.308490       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:00.311655       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:00.311863       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:00.312011       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:01.315425       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:01.315598       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:01.315723       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:02.318892       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:02.319076       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:02.319232       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:03.321788       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:03.321932       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:03.322014       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:04.325563       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:04.325683       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:04.325758       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:05.328577       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:05.328667       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:05.328708       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:06.332373       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:06.332504       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:06.332571       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:07.335858       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:07.336040       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:07.336184       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:08.339344       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:08.339505       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:08.340612       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:09.342951       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:09.343086       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:09.343142       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:10.346463       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:10.346594       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:10.346635       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:11.349836       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:11.350176       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:11.350379       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:12.353437       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:12.353573       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:12.353648       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:13.356306       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:13.356432       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:13.356494       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:14.359571       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:14.359697       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:14.359781       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:15.363557       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:15.363641       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:15.363713       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:16.367665       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:16.367820       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:16.367876       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:17.370854       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:17.370950       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:17.370994       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:18.375148       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:18.375329       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:18.375426       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:19.378650       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:19.378741       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:19.378797       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:20.382805       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:20.382923       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:20.382966       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:21.387202       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:21.387571       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:21.387693       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:22.390516       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:22.390708       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:22.390840       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:23.393682       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:23.393772       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:23.393812       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:24.397235       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:24.397458       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:24.397615       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:25.400965       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:25.401283       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:25.401420       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:26.403831       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:26.404859       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:26.404946       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:27.407703       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:27.407792       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:27.407834       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:28.411258       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:28.411417       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:28.411490       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:47:29Z" level=error msg="failed to save helper-pod-delete-pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
time="2025-04-28T02:47:29Z" level=error msg="failed to save helper-pod-delete-pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 02:47:29.414896       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:47:29Z" level=error msg="failed to save helper-pod-delete-pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916 logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 02:47:29.414965       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:29.415064       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:47:29Z" level=info msg="clean up volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c failed: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: create process timeout after 120 seconds" 
E0428 02:47:29.420421       1 controller.go:1558] "Volume deletion failed" err="failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: create process timeout after 120 seconds" PV="pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c"
I0428 02:47:29.420525       1 controller.go:1010] "Retrying syncing volume because failures < threshold" key="pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c" failures=7 threshold=15
E0428 02:47:29.420594       1 controller.go:1025] "Unhandled Error" err="error syncing volume \"pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c\": failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: create process timeout after 120 seconds" logger="UnhandledError"
I0428 02:47:29.420640       1 event.go:389] "Event occurred" object="pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: create process timeout after 120 seconds"
E0428 02:47:29.420953       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:47:29Z" level=info msg="clean up volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916 failed: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: create process timeout after 120 seconds" 
time="2025-04-28T02:47:29Z" level=info msg="clean up volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c failed: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: create process timeout after 120 seconds" 
E0428 02:47:29.422169       1 controller.go:1558] "Volume deletion failed" err="failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: create process timeout after 120 seconds" PV="pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916"
E0428 02:47:29.422199       1 controller.go:1558] "Volume deletion failed" err="failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: create process timeout after 120 seconds" PV="pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c"
I0428 02:47:29.422208       1 controller.go:1010] "Retrying syncing volume because failures < threshold" key="pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916" failures=7 threshold=15
I0428 02:47:29.422238       1 event.go:389] "Event occurred" object="pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: create process timeout after 120 seconds"
I0428 02:47:29.422250       1 controller.go:1010] "Retrying syncing volume because failures < threshold" key="pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c" failures=7 threshold=15
E0428 02:47:29.422299       1 controller.go:1025] "Unhandled Error" err="error syncing volume \"pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c\": failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: create process timeout after 120 seconds" logger="UnhandledError"
E0428 02:47:29.422240       1 controller.go:1025] "Unhandled Error" err="error syncing volume \"pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916\": failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: create process timeout after 120 seconds" logger="UnhandledError"
I0428 02:47:29.422341       1 event.go:389] "Event occurred" object="pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: create process timeout after 120 seconds"
E0428 02:47:29.425318       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:47:29.429185       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:52:26.858239       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:54:25.296926       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:54:47.976711       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:00.249443       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:00.258402       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:01.260567       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:02.263463       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:03.266592       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:04.269580       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:05.272811       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:06.276858       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:07.280398       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:08.283605       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:09.286596       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:10.289758       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:11.293491       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:12.297175       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:13.300876       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:14.304222       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:15.307397       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:16.310667       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:17.313529       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:18.316329       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:19.319296       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:20.321980       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:21.325638       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:22.329244       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:23.331933       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:24.335023       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:25.338558       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:26.344729       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:27.347592       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:28.350872       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:29.353779       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:30.356850       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:31.360491       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:32.363740       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:33.366330       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:34.369335       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:35.372137       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:36.376357       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:37.379317       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:38.382506       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:38.611142       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:38.619213       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:39.385833       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:39.621625       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:40.388474       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:40.624335       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:41.392533       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:41.627521       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:42.395543       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:42.629939       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:43.398192       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:43.633251       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:44.401639       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:44.636017       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:45.405966       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:45.639926       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:46.409410       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:46.643794       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:47.412949       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:47.646650       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:48.417291       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:48.649640       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:49.421194       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:49.653945       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:50.424184       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:50.656635       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:51.427737       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:51.659755       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:52.430962       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:52.663182       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:53.434175       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:53.666408       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:54.437324       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:54.669320       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:55.440483       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:55.672206       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:56.445423       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:56.675028       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:57.448146       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:57.677659       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:58.450972       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:58.681853       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:59.454136       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:56:59.684686       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:00.456717       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:00.688049       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:01.460111       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:01.690807       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:02.462755       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:02.694206       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:03.466030       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:03.697598       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:04.469296       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:04.702124       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:05.472363       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:05.705051       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:06.475105       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:06.707672       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:07.477779       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:07.710615       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:08.482477       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:08.713223       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:09.485407       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:09.716020       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:10.488095       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:10.719339       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:11.491357       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:11.721818       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:12.494245       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:12.725375       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:13.498017       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:13.509975       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:13.516794       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:13.728948       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:14.500995       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:14.519328       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:14.731803       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:15.507685       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:15.521493       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:15.734790       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:16.510788       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:16.524261       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:16.737481       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:17.513595       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:17.527145       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:17.740592       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:18.516240       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:18.529683       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:18.742941       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:19.519532       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:19.532186       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:19.747511       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:20.522523       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:20.535141       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:20.750465       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:21.525387       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:21.537894       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:21.753443       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:22.528495       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:22.540793       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:22.755704       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:23.530915       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:23.543521       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:23.758657       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:24.534199       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:24.545870       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:24.762127       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:25.537457       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:25.548490       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:25.764889       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:26.540783       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:26.551106       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:26.768452       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:27.543490       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:27.553298       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:27.771554       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:28.546772       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:28.556222       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:28.775042       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:29.549807       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:29.559944       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:29.778395       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:30.553824       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:30.562474       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:30.781787       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:31.557333       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:31.565032       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:31.785362       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:32.560555       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:32.567573       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:32.788407       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:33.564524       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:33.570999       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:33.792418       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:34.567683       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:34.573456       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:34.795863       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:35.571210       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:35.577129       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:35.798499       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:36.574752       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:36.579431       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:36.801597       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:37.578802       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:37.581436       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:37.805524       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:38.581628       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:38.583739       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:38.808479       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:39.584548       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:39.586694       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:39.811048       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:40.588606       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:40.588889       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:40.814244       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:41.592566       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:41.592698       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:41.817224       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:42.596717       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:42.596987       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:42.820613       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:43.601154       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:43.601343       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:43.824153       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:44.604036       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:44.604258       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:44.828277       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:45.607385       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:45.607580       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:45.831579       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:46.610537       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:46.610713       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:46.834759       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:47.613888       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:47.614078       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:47.837365       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:48.617278       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:48.617477       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:48.840634       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:49.620822       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:49.620971       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:49.844001       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:50.624620       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:50.624808       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:50.847181       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:51.628439       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:51.628625       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:51.850970       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:52.631992       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:52.632127       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:52.853259       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:53.635777       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:53.635968       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:53.856038       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:54.639286       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:54.639488       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:54.859415       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:55.642958       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:55.643117       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:55.862259       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:56.646394       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:56.647602       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:56.865068       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:57.649671       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:57.649841       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:57.868648       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:58.652887       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:58.653009       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:58.871368       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:59.656887       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:59.657057       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:57:59.875143       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:58:00Z" level=error msg="failed to save helper-pod-delete-pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916 logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 02:58:00.660168       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:00.660242       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:58:00Z" level=info msg="clean up volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916 failed: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: create process timeout after 120 seconds" 
E0428 02:58:00.665052       1 controller.go:1558] "Volume deletion failed" err="failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: create process timeout after 120 seconds" PV="pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916"
I0428 02:58:00.665133       1 controller.go:1010] "Retrying syncing volume because failures < threshold" key="pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916" failures=8 threshold=15
E0428 02:58:00.665199       1 controller.go:1025] "Unhandled Error" err="error syncing volume \"pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916\": failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: create process timeout after 120 seconds" logger="UnhandledError"
I0428 02:58:00.665253       1 event.go:389] "Event occurred" object="pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: create process timeout after 120 seconds"
E0428 02:58:00.665477       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:00.878666       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:01.664009       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:01.882105       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:02.667459       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:02.885830       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:03.670649       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:03.889676       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:04.674201       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:04.893515       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:05.677971       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:05.896864       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:06.681268       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:06.899381       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:07.684767       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:07.902486       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:08.689244       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:08.906330       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:09.692050       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:09.910137       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:10.695230       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:10.913570       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:11.699032       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:11.916442       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:12.703420       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:12.919502       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:13.706636       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:13.922144       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:14.711569       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:14.925721       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:15.714983       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:15.929045       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:16.718545       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:16.931756       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:17.721750       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:17.935145       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:18.725812       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:18.938533       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:19.729672       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:19.942862       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:20.732777       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:20.860511       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:20.947038       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:21.736206       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:21.950030       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:22.739365       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:22.953234       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:23.742424       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:23.956438       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:24.745724       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:24.959417       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:25.748768       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:25.963034       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:26.752457       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:26.965583       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:27.756211       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:27.969403       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:28.759343       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:28.972479       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:29.763061       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:29.976082       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:30.766299       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:30.979486       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:31.769746       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:31.982686       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:32.773621       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:32.985576       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:33.778002       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:33.989138       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:34.785450       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:34.992406       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:35.789326       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:35.994872       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:36.803823       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:36.997831       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:37.806959       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:38.000662       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:38.810258       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:58:39Z" level=error msg="failed to save helper-pod-delete-pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 02:58:39.004494       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:58:39Z" level=info msg="clean up volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c failed: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: create process timeout after 120 seconds" 
E0428 02:58:39.008726       1 controller.go:1558] "Volume deletion failed" err="failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: create process timeout after 120 seconds" PV="pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c"
I0428 02:58:39.008778       1 controller.go:1010] "Retrying syncing volume because failures < threshold" key="pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c" failures=8 threshold=15
E0428 02:58:39.008823       1 controller.go:1025] "Unhandled Error" err="error syncing volume \"pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c\": failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: create process timeout after 120 seconds" logger="UnhandledError"
I0428 02:58:39.008942       1 event.go:389] "Event occurred" object="pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: create process timeout after 120 seconds"
E0428 02:58:39.009175       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:39.813807       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:40.817172       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:41.821357       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:42.824392       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:43.827240       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:44.830237       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:45.833156       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:46.836896       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:47.840243       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:48.843206       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:49.846357       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:50.849085       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:51.852579       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:52.855616       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:53.858473       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:54.861343       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:55.864112       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:56.867258       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:57.870225       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:58.873478       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:58:59.876547       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:59:00.880144       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:59:01.883375       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:59:02.886469       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:59:03.890209       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:59:04.893613       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:59:05.898175       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:59:06.901042       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:59:07.903831       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:59:08.906730       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:59:09.909668       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:59:10.913121       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:59:11.917377       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 02:59:12.920793       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:59:13Z" level=error msg="failed to save helper-pod-delete-pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 02:59:13.924679       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T02:59:13Z" level=info msg="clean up volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c failed: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: create process timeout after 120 seconds" 
E0428 02:59:13.930656       1 controller.go:1558] "Volume deletion failed" err="failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: create process timeout after 120 seconds" PV="pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c"
I0428 02:59:13.930728       1 controller.go:1010] "Retrying syncing volume because failures < threshold" key="pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c" failures=8 threshold=15
E0428 02:59:13.930788       1 controller.go:1025] "Unhandled Error" err="error syncing volume \"pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c\": failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: create process timeout after 120 seconds" logger="UnhandledError"
I0428 02:59:13.930848       1 event.go:389] "Event occurred" object="pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: create process timeout after 120 seconds"
E0428 02:59:13.931084       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:28.974730       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:28.974820       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:28.974890       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:28.982247       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:28.983844       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:28.984162       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:29.986528       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:29.986645       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:29.986729       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:30.989596       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:30.989730       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:30.989828       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:31.992858       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:31.992972       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:31.993023       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:32.996044       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:32.996129       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:32.996171       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:33.999565       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:33.999677       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:33.999766       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:35.002752       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:35.002864       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:35.002931       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:36.006456       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:36.006610       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:36.006747       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:37.009641       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:37.009878       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:37.009981       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:38.013070       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:38.013235       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:38.013285       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:39.016588       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:39.016677       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:39.016721       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:40.020157       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:40.020253       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:40.020300       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:41.023566       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:41.023656       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:41.023698       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:42.027698       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:42.027799       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:42.027847       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:43.031390       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:43.031548       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:43.031606       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:44.034708       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:44.034792       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:44.034835       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:45.037386       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:45.037498       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:45.037545       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:46.041601       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:46.041694       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:46.041739       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:47.046453       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:47.046564       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:47.046620       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:48.050804       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:48.050893       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:48.050944       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:49.054479       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:49.054650       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:49.054698       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:50.058620       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:50.058766       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:50.058812       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:51.063568       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:51.063882       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:51.064056       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:52.067493       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:52.067632       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:52.067699       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:53.071969       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:53.072126       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:53.072166       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:54.075708       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:54.075858       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:54.075967       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:55.079608       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:55.079767       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:55.079847       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:56.083103       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:56.083271       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:56.083402       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:57.086710       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:57.086797       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:57.086838       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:58.090188       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:58.090348       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:58.090437       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:59.093661       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:59.093791       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:00:59.093939       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:00.097775       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:00.097863       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:00.097939       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:01.101582       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:01.101675       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:01.101720       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:02.105147       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:02.105266       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:02.105324       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:03.108549       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:03.108663       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:03.108726       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:04.111527       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:04.111662       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:04.111729       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:05.115261       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:05.115531       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:05.115741       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:06.118799       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:06.118984       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:06.119099       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:07.123219       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:07.123404       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:07.123558       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:08.126675       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:08.126933       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:08.127079       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:09.130199       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:09.130529       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:09.130620       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:10.135206       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:10.135389       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:10.135435       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:11.139240       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:11.139550       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:11.139715       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:12.142779       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:12.143011       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:12.143090       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:13.146037       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:13.146121       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:13.146161       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:14.149130       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:14.149241       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:14.149340       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:15.152165       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:15.152249       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:15.152291       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:16.155842       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:16.155950       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:16.156003       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:17.160218       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:17.160329       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:17.160390       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:18.163460       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:18.163644       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:18.163712       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:19.166641       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:19.166758       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:19.166951       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:20.170350       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:20.170614       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:20.170761       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:21.173605       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:21.173726       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:21.173780       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:22.177227       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:22.177422       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:22.177548       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:23.180339       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:23.180433       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:23.180488       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:24.183854       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:24.183976       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:24.184022       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:25.187717       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:25.187896       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:25.188061       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:26.191449       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:26.191663       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:26.191756       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:27.194605       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:27.194767       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:27.194804       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:28.197857       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:28.197975       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:28.198042       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:29.200786       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:29.200888       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:29.200960       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:30.204715       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:30.204890       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:30.205066       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:31.209212       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:31.209325       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:31.209395       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:32.212542       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:32.212623       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:32.212668       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:33.216018       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:33.216191       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:33.216342       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:34.219453       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:34.219641       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:34.219751       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:35.222880       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:35.223069       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:35.223202       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:36.226900       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:36.227051       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:36.227178       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:37.230625       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:37.230789       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:37.230955       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:38.233690       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:38.233829       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:38.233959       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:39.236968       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:39.237057       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:39.237096       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:40.240629       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:40.240817       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:40.240967       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:41.243986       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:41.244194       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:41.244302       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:42.247040       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:42.247236       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:42.247360       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:43.250541       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:43.250643       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:43.250685       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:44.253878       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:44.254003       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:44.254055       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:45.257735       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:45.257890       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:45.258005       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:46.260806       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:46.260973       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:46.261035       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:47.264516       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:47.264640       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:47.264711       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:48.267796       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:48.267964       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:48.268054       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:49.271837       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:49.272089       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:49.272305       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:50.275395       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:50.275505       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:50.275578       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:51.278569       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:51.278737       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:51.278854       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:52.282436       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:52.282679       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:52.282853       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:53.286829       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:53.286981       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:53.287052       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:54.290095       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:54.290204       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:54.290282       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:55.293703       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:55.293950       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:55.294110       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:56.297856       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:56.297962       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:56.298020       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:57.301125       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:57.301322       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:57.301486       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:58.304037       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:58.304217       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:58.304369       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:59.307988       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:59.308222       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:01:59.308399       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:00.311851       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:00.312062       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:00.312267       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:01.314830       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:01.314998       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:01.315070       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:02.318041       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:02.318170       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:02.318246       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:03.321271       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:03.321461       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:03.321596       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:04.324507       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:04.324609       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:04.324658       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:05.328059       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:05.328235       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:05.328369       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:06.331709       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:06.331877       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:06.332023       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:07.334862       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:07.335027       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:07.335113       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:08.338123       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:08.338215       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:08.338291       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:09.341588       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:09.341718       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:09.341778       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:10.345147       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:10.345344       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:10.345418       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:11.348588       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:11.348712       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:11.348761       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:12.352252       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:12.352416       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:12.352463       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:13.355637       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:13.355780       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:13.355846       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:14.359174       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:14.359307       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:14.359385       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:15.362007       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:15.362204       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:15.362296       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:16.364964       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:16.365050       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:16.365112       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:17.368732       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:17.368980       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:17.369074       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:18.372109       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:18.372214       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:18.372284       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:18.979593       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:19.375760       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:19.375885       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:19.375977       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:20.379749       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:20.379854       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:20.379953       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:21.382781       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:21.383782       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:21.383874       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:22.385806       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:22.386097       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:22.386181       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:23.388774       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:23.388876       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:23.388969       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:24.392534       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:24.392621       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:24.392682       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:25.396498       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:25.396705       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:25.396864       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:26.399541       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:26.399659       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:26.399747       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:27.402704       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:27.402843       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:27.402958       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:28.405951       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:28.406080       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:28.406155       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T03:02:29Z" level=error msg="failed to save helper-pod-delete-pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
time="2025-04-28T03:02:29Z" level=error msg="failed to save helper-pod-delete-pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916 logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
time="2025-04-28T03:02:29Z" level=error msg="failed to save helper-pod-delete-pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 03:02:29.409752       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:29.409839       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:29.409931       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T03:02:29Z" level=info msg="clean up volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c failed: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: create process timeout after 120 seconds" 
E0428 03:02:29.415098       1 controller.go:1558] "Volume deletion failed" err="failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: create process timeout after 120 seconds" PV="pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c"
time="2025-04-28T03:02:29Z" level=info msg="clean up volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c failed: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: create process timeout after 120 seconds" 
E0428 03:02:29.415140       1 controller.go:1558] "Volume deletion failed" err="failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: create process timeout after 120 seconds" PV="pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c"
I0428 03:02:29.415175       1 controller.go:1010] "Retrying syncing volume because failures < threshold" key="pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c" failures=9 threshold=15
I0428 03:02:29.415187       1 event.go:389] "Event occurred" object="pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: create process timeout after 120 seconds"
I0428 03:02:29.415143       1 controller.go:1010] "Retrying syncing volume because failures < threshold" key="pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c" failures=9 threshold=15
time="2025-04-28T03:02:29Z" level=info msg="clean up volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916 failed: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: create process timeout after 120 seconds" 
E0428 03:02:29.415238       1 controller.go:1025] "Unhandled Error" err="error syncing volume \"pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c\": failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: create process timeout after 120 seconds" logger="UnhandledError"
E0428 03:02:29.415245       1 controller.go:1025] "Unhandled Error" err="error syncing volume \"pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c\": failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: failed to delete volume pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c: create process timeout after 120 seconds" logger="UnhandledError"
E0428 03:02:29.415259       1 controller.go:1558] "Volume deletion failed" err="failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: create process timeout after 120 seconds" PV="pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916"
I0428 03:02:29.415275       1 event.go:389] "Event occurred" object="pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: failed to delete volume pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c: create process timeout after 120 seconds"
I0428 03:02:29.415350       1 controller.go:1010] "Retrying syncing volume because failures < threshold" key="pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916" failures=9 threshold=15
E0428 03:02:29.415403       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
I0428 03:02:29.415418       1 event.go:389] "Event occurred" object="pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: create process timeout after 120 seconds"
E0428 03:02:29.415444       1 controller.go:1025] "Unhandled Error" err="error syncing volume \"pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916\": failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: failed to delete volume pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916: create process timeout after 120 seconds" logger="UnhandledError"
E0428 03:02:29.419186       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:02:29.423023       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:04:01.299067       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:04:17.863067       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:09:03.982154       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:12:08.301249       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:13:19.865207       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:14:40.670479       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:14:40.681158       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:14:41.684335       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:14:42.687045       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:14:43.690332       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:14:44.693300       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:14:45.696665       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:14:46.700225       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:14:47.703368       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:14:48.706595       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:14:49.709844       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T03:14:49Z" level=error msg="failed to save helper-pod-delete-pvc-49b67dfa-6e0f-41de-ba47-1c07a6f7b916 logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 03:14:49.713302       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:14:49.720621       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:15:19.011604       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:15:19.019310       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:15:20.021381       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:15:21.024760       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:15:22.028463       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T03:15:22Z" level=error msg="failed to save helper-pod-delete-pvc-035a7b6c-e4d8-493e-944f-022fd5179d4c logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 03:15:22.031508       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:15:22.037338       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:15:28.976883       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:15:28.981637       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:15:29.984002       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:15:30.987569       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:15:31.990658       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T03:15:31Z" level=error msg="failed to save helper-pod-delete-pvc-2064cbfe-0c8e-49f4-90f7-4e1fd94c117c logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 03:15:31.993502       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:15:31.998979       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:17:03.984273       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:20:19.303961       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:20:48.867135       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:23:18.987200       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:25:58.306583       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:29:41.869971       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:30:30.989286       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:32:13.309047       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:35:47.991782       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:37:06.871841       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:41:01.994278       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:41:22.311608       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:44:40.874399       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:46:36.313552       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:50:26.996563       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:53:44.315592       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:54:05.876746       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 03:57:54.999110       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:00:04.878708       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:02:29.318004       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:06:05.880812       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:06:33.001529       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:08:41.320857       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:41.024689       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:41.046584       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:41.046699       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:41.065467       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:41.507683       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:41.510762       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:41.511116       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:41.515955       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:42.073888       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:42.117746       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:42.123668       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:42.123868       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:42.128679       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:42.518960       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:43.083583       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:43.131723       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:43.532064       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:44.087049       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:44.137659       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:44.536240       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:45.092559       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:45.143158       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:45.541340       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:46.106442       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T04:11:46Z" level=error msg="failed to save helper-pod-create-pvc-7083f87e-fa6f-44ae-a687-74ca847918bf logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 04:11:46.120453       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:46.136006       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:46.149657       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:46.684139       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:46.684328       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:47.954897       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:47.955073       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:48.959521       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:48.959755       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T04:11:48Z" level=error msg="failed to save helper-pod-create-pvc-67765f39-b6f9-4f29-8912-dc21af9cacca logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 04:11:48.968866       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T04:11:48Z" level=error msg="failed to save helper-pod-create-pvc-c2bf1c25-c551-4322-98b6-76ead14ca30c logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 04:11:48.969286       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:48.976987       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:48.978990       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:48.991816       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:11:48.999143       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:13:32.008043       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:14:25.882832       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:17:03.323936       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:20:03.010371       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:23:42.885760       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:25:39.012687       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:26:21.326141       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:31:09.887817       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:33:44.014616       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:35:02.328427       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:37:00.890382       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:39:46.017391       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:40:58.331188       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:42:18.892758       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:44:47.020043       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:48:47.334032       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:49:18.895791       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:50:45.022292       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:55:35.336605       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:58:25.897590       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 04:58:36.024505       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:02:56.338803       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:03:39.026524       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:03:53.899386       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:10:10.901212       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:10:17.029305       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:10:50.341110       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:17:24.343290       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:17:59.903183       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:19:30.031530       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:23:13.905745       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:24:33.033844       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:26:51.345857       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:30:46.908333       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:33:24.036128       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:34:34.348141       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:38:28.911125       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:38:34.038381       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:41:38.350817       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:47:16.040241       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:47:52.919527       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:48:01.074217       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:48:01.080598       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:48:02.082652       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:48:03.085327       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:48:04.088004       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T05:48:04Z" level=error msg="failed to save helper-pod-delete-pvc-c2bf1c25-c551-4322-98b6-76ead14ca30c logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 05:48:04.092473       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:48:04.100703       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:48:04.441330       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:48:04.445048       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:48:05.221625       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:48:05.226267       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:48:05.449045       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:48:06.229457       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:48:06.452474       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T05:48:06Z" level=error msg="failed to save helper-pod-delete-pvc-7083f87e-fa6f-44ae-a687-74ca847918bf logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 05:48:06.454949       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:48:06.460843       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:48:07.232566       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:48:08.235708       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T05:48:08Z" level=error msg="failed to save helper-pod-delete-pvc-67765f39-b6f9-4f29-8912-dc21af9cacca logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 05:48:08.238089       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:48:08.245294       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:51:32.353716       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:53:48.043153       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:54:13.925855       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 05:57:24.355544       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:01:36.045792       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:02:22.927829       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:05:57.358444       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:07:50.930005       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:09:09.048382       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:12:49.360806       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:16:49.932643       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:17:06.051026       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:18:04.363477       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:22:45.053986       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:23:19.935382       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:25:18.366439       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:29:59.938228       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:30:25.369710       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:30:53.056651       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:35:37.372044       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:37:19.940602       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:40:16.059375       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:42:02.374711       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:44:46.754184       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:44:46.760016       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:44:46.760176       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:44:46.768234       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:44:47.770779       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:44:47.978467       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:44:47.981924       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:44:47.982225       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:44:47.986409       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:44:48.779634       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:44:48.988810       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:44:49.868945       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T06:44:49Z" level=error msg="failed to save helper-pod-create-pvc-72c52dc8-47f1-459b-a9ae-c6f40ab80454 logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 06:44:49.873585       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:44:49.893652       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:44:49.898387       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:44:50.006524       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:44:51.012142       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T06:44:51Z" level=error msg="failed to save helper-pod-create-pvc-6d781704-9c43-47ec-954d-7725c3e3afef logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 06:44:51.014852       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:44:51.022986       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:44:51.026039       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:46:46.943093       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:47:17.376679       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:49:08.063419       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:49:08.089134       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:49:08.089504       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:49:08.174122       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:49:09.233163       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:49:10.241149       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:49:11.245261       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T06:49:11Z" level=error msg="failed to save helper-pod-create-pvc-ccf1e15b-f3de-491c-a1b3-80285ec30af9 logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 06:49:11.248489       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:49:11.256211       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:49:11.259708       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:50:06.061764       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:51:50.945498       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:52:30.378645       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:57:55.948059       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:58:27.064531       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 06:59:00.381447       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:04:21.384338       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:07:33.949870       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:08:23.067014       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:14:13.386791       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:16:39.952526       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:17:57.069477       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:22:18.389085       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:23:13.955268       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:25:11.072511       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:28:02.694150       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:28:02.697378       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:28:02.698007       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:28:02.706480       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:28:03.709090       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:28:04.711774       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:28:05.714236       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:28:06.718230       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:28:07.722160       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:28:08.725721       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:28:09.728361       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:28:10.731783       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:28:11.734621       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:28:12.738257       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
time="2025-04-28T07:28:12Z" level=error msg="failed to save helper-pod-create-pvc-90cc31d6-3f39-4553-8947-ee6ce2977f70 logs: unable to retrieve in cluster config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" 
E0428 07:28:12.741208       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:28:12.746748       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:28:12.750379       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:28:18.957788       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:31:36.399130       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:32:57.074475       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:34:11.960317       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:38:10.409328       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:40:54.963247       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:41:48.076707       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:44:08.411315       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:48:35.078722       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:49:57.965135       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:51:58.413415       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:56:43.082024       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:57:06.415384       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 07:58:40.967779       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:04:59.084593       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:05:22.417353       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:08:06.971096       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:11:27.087274       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:14:46.420199       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:14:51.973480       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:17:20.089239       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:20:22.975225       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:22:51.422707       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:27:06.092161       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:29:21.977569       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:31:58.424581       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:33:06.094588       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:36:02.980689       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:37:43.427363       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:39:46.097050       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:42:55.983385       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:43:27.430188       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:46:45.100396       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:48:32.432659       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:50:50.986055       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:52:03.103022       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 08:54:25.434944       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:00:04.988798       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:00:35.105774       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:01:17.436873       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:08:01.439149       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:09:26.991221       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:10:25.107850       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:15:44.444055       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:16:17.993243       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:16:49.110044       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:21:56.112464       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:22:42.446630       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:23:56.995681       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:29:23.449113       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:31:03.114208       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:33:26.998402       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:36:59.451357       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:38:57.116094       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:41:43.000647       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:42:16.453180       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:47:09.119000       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:48:06.002844       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:51:19.455265       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:52:50.121853       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:55:31.005606       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:59:00.458078       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 09:59:28.124266       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:02:36.008386       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:05:23.127022       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:06:18.460482       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:10:39.010948       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:11:04.129592       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:15:36.463627       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:15:47.013456       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:19:44.132695       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:22:22.016318       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:25:13.466344       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:25:49.134630       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:30:27.018189       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:32:42.468881       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:34:36.137825       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:37:17.020335       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:40:11.471076       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:41:08.140624       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:44:07.022708       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:48:34.473110       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:50:39.143253       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:53:31.024751       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:54:52.475520       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:58:51.028110       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 10:59:12.145433       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:03:27.477735       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:05:19.147854       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:05:34.030827       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:10:36.033047       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:10:56.479956       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:12:30.150865       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:19:15.035547       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:20:18.153199       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:20:22.481728       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:25:37.483529       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:25:59.038164       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:26:10.155187       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:33:13.158002       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:34:25.040203       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:34:27.485619       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:40:37.488091       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:41:38.160790       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:42:58.042169       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:45:54.489843       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:48:37.044133       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:50:54.492042       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:51:32.163949       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:54:52.047068       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:57:39.494878       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 11:59:36.166226       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:02:40.049171       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:02:44.497221       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:09:31.169525       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:09:47.051969       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:11:47.498703       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:17:34.501489       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:17:51.172227       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:19:06.054988       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:25:43.174686       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:25:47.503570       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:29:04.057538       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:35:21.177224       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:35:27.060245       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:35:30.505201       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:40:46.506946       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:40:58.062413       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:41:51.179987       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:46:53.509739       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:48:34.182548       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:48:47.065245       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:54:24.067590       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:56:02.512052       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 12:56:52.184217       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:02:43.070962       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:03:47.186433       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:05:50.514445       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:08:04.074245       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:09:04.188529       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:12:00.517592       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:16:39.191123       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:16:46.076341       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:19:21.520171       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:22:16.078058       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:23:29.193489       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:26:47.522382       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:29:44.080084       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:33:14.195591       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:34:29.524309       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:37:39.082978       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:39:54.197966       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:43:04.526780       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:47:12.086162       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:47:23.201614       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:48:11.529054       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:52:54.203805       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:53:29.531210       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 13:55:17.088075       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:02:01.534290       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:02:13.206361       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:03:27.090232       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:07:26.208686       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:07:33.536715       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:09:28.092267       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:14:39.539089       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:15:59.094086       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:16:23.211222       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:22:17.097329       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:22:31.541704       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:22:55.213552       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:29:31.100374       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:29:59.216015       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:31:04.543891       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:35:02.102597       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:36:04.546244       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:36:25.218486       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:41:26.549283       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:42:18.104900       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:45:20.220739       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:48:19.551399       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:48:37.107070       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:53:06.223440       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:54:17.109277       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 14:58:01.553170       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:01:14.226396       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:01:28.111469       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:06:36.114468       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:06:50.228560       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:06:51.555226       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:14:26.557228       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:14:30.230526       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:15:03.116445       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:22:14.559365       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:22:56.231950       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:23:29.119184       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:29:11.561347       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:31:55.121086       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:32:34.234600       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:36:51.563350       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:38:21.124186       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:41:37.237139       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:41:56.566055       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:44:59.126131       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:47:09.239964       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:47:28.568771       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:53:11.571705       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:53:29.244368       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:54:05.128507       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 15:59:03.247216       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:00:52.130445       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:01:53.573714       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:05:20.249623       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:06:54.575438       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:08:17.133289       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:13:21.252707       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:14:44.577485       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:18:03.135445       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:19:18.255095       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:24:32.580654       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:25:21.137625       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:25:45.257026       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:33:22.260500       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:33:53.139549       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:34:27.583262       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:40:48.141942       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:43:12.263089       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:43:18.585901       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:46:55.144679       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:49:27.587874       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:51:17.265122       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:52:44.146550       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:57:15.590720       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:59:13.148534       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 16:59:58.266995       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:04:42.593482       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:05:00.150928       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:09:38.269120       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:13:00.595617       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:13:45.153454       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:16:13.271949       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:20:40.156435       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:21:22.598324       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:25:09.275032       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:29:57.158841       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:31:18.600869       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:33:38.277703       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:38:10.160588       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:40:08.602979       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:41:37.280466       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:43:31.163115       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:47:41.282685       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:49:21.605238       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:52:04.165884       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:56:21.285417       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 17:58:58.607675       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:00:07.168230       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:03:29.287892       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:07:01.609682       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:07:53.170664       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:12:50.290544       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:14:17.613242       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:16:08.173208       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:20:49.293236       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:24:02.176058       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:24:11.615020       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:28:59.295959       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:32:50.178107       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:33:42.616494       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:35:06.298397       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:41:20.179952       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:42:17.618256       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:44:18.300530       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:47:07.181574       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:49:32.303495       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:50:23.620726       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:54:15.183737       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:58:10.305496       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 18:59:32.622949       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:03:16.185772       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:06:09.308747       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:08:06.625017       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:10:46.188079       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:11:40.310734       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:17:42.313841       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:17:45.627465       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:18:52.190398       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:23:59.316266       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:24:37.193407       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:25:58.629099       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:31:24.318738       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:34:34.196381       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:35:48.631796       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:41:14.321300       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:41:26.633852       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:42:33.198899       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:48:47.636132       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:49:10.323036       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:51:38.201710       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:56:13.639176       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:58:53.325965       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 19:59:52.203670       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:03:23.641853       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:05:20.329217       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:08:47.206286       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:09:48.644204       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:14:43.331423       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:15:26.646404       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:15:45.208899       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:22:32.334347       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:22:40.211646       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:23:34.649457       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:28:28.337242       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:28:36.652128       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:31:44.213887       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:37:08.653830       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:38:03.216566       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:38:10.339558       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:45:21.218747       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:46:42.655777       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:47:08.341357       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:50:58.220832       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:55:37.658694       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:55:43.343602       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 20:56:31.223388       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:04:23.661248       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:04:55.345293       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:04:59.225342       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:10:56.355405       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:11:28.662798       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:12:23.227438       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:18:16.665947       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:19:40.230096       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:19:40.358182       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:23:40.668680       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:26:24.360756       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:27:18.232343       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:29:55.671048       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:31:35.362585       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:34:27.234371       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:38:42.674121       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:39:22.364520       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:41:23.236555       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:46:28.367019       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:48:08.676841       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:48:15.239332       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:54:08.369474       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:54:55.678791       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 21:55:08.242039       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:00:55.245222       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:01:16.371788       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:03:47.681448       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:07:17.374701       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:10:13.247744       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:10:35.683729       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:13:12.376691       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:16:02.250373       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:18:56.686449       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:21:02.378834       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:24:03.252464       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:27:09.688658       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:28:55.381734       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:32:22.690713       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:32:48.255063       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:38:25.383985       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:40:39.257062       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:41:53.692649       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:47:28.387111       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:48:06.695306       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:48:39.259592       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:52:28.389271       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:55:19.262436       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 22:55:41.698395       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:01:19.392428       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:01:23.264995       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:02:57.700866       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:08:37.267426       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:10:43.395288       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:11:00.703203       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:16:27.397528       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:17:38.705987       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:18:00.269693       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:24:07.400523       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:25:04.708179       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:27:39.272628       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:30:48.403666       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:34:08.710703       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:34:43.275124       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:38:07.406155       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:40:27.279298       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:40:50.713707       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:47:20.715953       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:47:58.408305       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:50:01.282130       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:54:31.717856       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:55:06.410362       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0428 23:55:49.284439       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:02:19.413596       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:03:05.287028       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:04:24.720308       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:07:19.415433       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:08:53.289442       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:12:26.723077       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:15:35.417370       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:18:02.291874       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:18:03.725453       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:23:40.727468       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:24:58.419526       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:26:45.293670       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:30:07.421542       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:31:46.296423       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:32:31.729361       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:39:24.424290       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:39:29.731217       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:41:25.299471       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:46:45.732647       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:48:20.426014       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:49:33.302171       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:54:37.303899       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:54:51.735404       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 00:55:45.428141       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:01:37.430157       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:02:27.737147       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:03:34.306805       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:07:39.432873       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:10:40.741246       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:11:33.308615       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:15:46.743981       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:15:47.434667       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:19:10.310994       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:21:24.746492       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:22:20.436848       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:27:17.312684       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:28:41.438931       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:29:03.749136       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:34:45.751351       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:34:50.315101       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:37:19.443417       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:42:57.317703       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:43:48.445698       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:44:32.753993       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:48:59.448800       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:49:07.320557       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:53:26.756671       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:56:01.450681       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 01:56:11.322135       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:02:07.759192       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:03:23.453143       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:05:08.324195       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:10:17.456231       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:11:33.761250       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:14:00.326676       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:16:12.458029       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:17:42.763777       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:23:11.329238       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:23:15.459954       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:25:36.767295       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:30:22.461419       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:32:03.331480       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:32:12.770423       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:36:09.464228       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:37:59.772896       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:40:17.333534       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:43:45.466286       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:44:59.775648       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:45:27.335495       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:50:39.777884       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:50:51.468097       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:51:46.338248       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:58:20.470026       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 02:59:42.780273       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:00:11.340931       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:03:59.472491       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:06:31.343587       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:08:06.782578       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:10:48.475063       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:15:30.346378       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:16:09.785132       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:18:51.477732       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:23:58.480215       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:24:12.787289       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:25:20.349043       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:31:33.482877       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:33:44.789355       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:34:59.351653       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:38:51.485346       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:39:40.791788       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:44:17.354434       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:46:23.487777       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:49:20.793969       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:53:03.356605       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:54:32.490233       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:55:28.796776       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 03:59:10.359800       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:03:11.798831       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:03:24.491804       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:07:00.362969       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:08:31.800977       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:12:39.493619       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:13:53.365235       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:16:10.803164       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:21:20.367322       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:22:07.496327       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:22:57.805337       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:27:35.499291       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:30:07.370033       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:30:28.807484       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:33:19.502097       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:36:14.810043       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:39:33.372497       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:42:34.812840       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:43:16.504789       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:49:12.506932       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:49:14.374682       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:51:44.814726       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:54:21.376475       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 04:56:44.509696       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:01:13.818102       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:01:28.378644       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:03:25.511934       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:07:03.820057       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:07:05.380967       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:10:20.514964       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:12:25.822701       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:13:22.382219       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:16:00.517999       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:18:55.385017       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:19:23.825336       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:25:20.520435       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:27:44.827958       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:28:41.387387       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:33:05.523410       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:34:34.830330       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:37:43.395747       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:41:29.526173       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:43:31.833011       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:45:08.399229       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:50:07.835362       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:50:17.529092       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:51:18.401822       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:56:06.836943       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:56:30.403781       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 05:59:33.530796       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:03:52.406544       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:04:51.532761       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:05:53.839284       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:10:59.841756       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:11:28.535135       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:12:11.409009       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:17:48.537511       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:18:13.844071       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:19:26.411931       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:25:09.846847       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:25:28.540247       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:27:53.414952       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:31:55.850269       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:34:20.542751       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:37:13.417492       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:38:02.852716       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:41:44.545756       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:43:21.855460       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:44:32.419963       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:48:52.548674       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:51:26.421680       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:51:53.857794       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:55:47.550988       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:56:45.424521       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 06:59:43.860516       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:01:36.552706       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:06:08.427022       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:09:32.863529       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:10:44.555323       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:13:59.429660       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:15:59.867381       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:16:04.557130       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:22:21.432036       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:25:22.559435       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:25:34.870084       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:30:51.435139       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:33:23.561253       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:35:25.872574       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:39:45.564561       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:40:45.437135       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:42:36.875351       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:47:36.567096       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:48:30.439751       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:52:28.877720       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:54:47.443992       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:56:39.569336       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 07:58:38.879416       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:02:28.446251       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:03:33.572268       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:06:30.882325       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:11:33.449118       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:11:57.574400       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:13:31.884944       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:17:34.576389       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:19:43.451638       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:20:16.886878       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:26:12.579009       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:27:57.453873       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:28:33.888875       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:34:00.582260       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:35:07.891558       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:35:42.456273       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:42:15.584308       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:43:19.893746       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:44:18.458009       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:49:07.896014       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:49:29.586088       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:51:02.460432       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:55:11.588399       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:56:39.463256       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 08:58:52.897748       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:04:26.465634       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:04:43.591789       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:05:05.899873       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:09:46.468210       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:13:24.902465       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:14:22.594006       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:18:32.470576       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:21:35.596355       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:22:22.904524       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:26:47.473067       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:28:09.906726       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:28:27.598700       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:34:25.600466       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:35:52.475943       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:36:32.908835       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:42:27.602960       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:43:19.911033       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:44:25.478277       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:49:17.913658       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:50:48.605459       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:50:54.480689       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:56:39.607483       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:56:48.916593       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 09:57:36.483261       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:02:26.918168       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:02:55.485217       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:05:46.609700       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:07:53.920280       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:11:22.487510       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:13:14.612605       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:15:24.921992       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:19:39.490605       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:21:40.924426       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:22:12.615093       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:26:09.493355       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:29:34.926634       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:30:08.617838       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:31:31.495564       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:34:38.928766       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:38:01.498572       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:39:24.620764       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:41:36.930815       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:44:33.501223       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:46:08.622815       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:47:27.933258       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:50:04.503149       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:52:33.936205       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:54:06.624814       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 10:59:03.505361       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:01:26.626502       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:01:40.938962       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:07:24.508498       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:09:55.629591       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:10:28.941834       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:14:13.510631       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:17:11.944550       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:18:22.631616       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:20:29.512804       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:24:33.633384       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:26:07.946586       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:30:21.514795       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:31:26.635779       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:31:41.948618       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:35:34.517641       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:37:53.639094       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:38:14.951018       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:42:50.519764       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:44:43.641646       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:46:56.953230       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:48:00.521799       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:54:08.644032       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:54:16.955389       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:57:59.523953       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 11:59:21.956980       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:01:06.646488       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:07:41.649560       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:07:55.959737       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:07:57.526506       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:15:53.961981       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:16:18.651675       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:16:27.527729       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:21:53.964646       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:22:03.654307       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:26:23.530277       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:30:59.655846       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:31:32.967451       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:35:26.533095       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:39:47.658639       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:40:30.969315       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:42:47.535239       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:45:54.660570       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:49:36.974159       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:50:29.537365       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:53:35.662401       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:55:43.976734       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:55:58.539611       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 12:58:59.664470       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:02:26.541997       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:02:54.978434       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:06:53.667141       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:09:34.981365       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:11:51.544761       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:16:07.670081       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:17:14.546737       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:17:25.983620       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:23:29.673181       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:23:57.548607       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:23:58.986111       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:28:35.675632       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:29:45.551211       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:33:49.988602       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:35:08.677309       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:39:32.554097       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:40:31.990886       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:41:57.680440       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:47:38.556474       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:49:31.683168       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:49:39.993412       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:56:11.558978       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:57:26.685937       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 13:57:36.995107       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:03:51.561029       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:05:43.688406       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:07:08.997580       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:12:29.563317       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:14:48.690741       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:16:54.000467       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:21:05.565918       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:22:22.003688       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:24:22.693153       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:29:41.568133       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:31:30.695374       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:32:01.006363       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:35:14.571203       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:38:49.697317       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:40:16.573650       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:40:24.008997       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:45:40.700267       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:46:32.575540       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:48:25.010395       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:54:37.702893       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:54:43.577254       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:55:37.013135       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 14:59:50.579388       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:02:06.705143       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:02:09.014733       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:06:21.581662       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:07:10.707989       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:08:37.016469       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:13:45.584346       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:15:58.709730       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:17:50.019141       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:23:03.587138       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:24:04.711607       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:24:23.021483       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:28:38.596341       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:30:06.023706       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:32:28.713513       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:34:11.598565       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:35:51.025739       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:38:00.716217       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:41:03.601469       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:41:15.028216       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:44:32.718840       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:47:29.604212       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:49:27.030731       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:51:01.721441       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:54:44.032802       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:54:53.606708       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 15:57:03.723325       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:01:47.608888       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:02:51.725126       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:03:49.035198       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:10:45.611562       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:12:26.727309       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:13:29.037426       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:16:16.613958       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:20:45.729807       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:20:55.039241       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:24:03.616037       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:29:41.731987       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:29:51.041383       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:33:04.618228       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:35:27.734020       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:38:32.043301       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:41:37.621413       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:43:52.735721       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:46:23.045706       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:48:37.623319       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:52:16.738111       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:53:50.626190       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:54:09.048339       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 16:58:52.739900       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:02:46.628439       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:03:04.051160       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:05:03.742256       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:10:31.054047       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:10:48.630718       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:14:16.745068       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:15:53.056487       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:18:41.633754       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:22:35.747357       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:24:07.059453       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:27:28.636941       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:29:19.062129       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:31:05.749780       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:33:47.639233       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:35:34.064703       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:37:39.752221       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:41:25.641329       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:44:50.067396       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:47:09.754837       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:48:26.643125       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:51:41.069317       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:54:26.645395       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:55:38.756944       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 17:58:18.071623       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:03:57.647027       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:04:30.758963       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:06:53.073540       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:10:10.650580       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:13:01.075651       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:13:33.761957       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:19:55.653605       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:20:48.077737       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:21:11.763817       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:25:55.656481       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:30:03.767049       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:30:47.079618       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:35:08.769753       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:35:32.658864       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:38:04.082467       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:43:37.661331       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:44:02.772398       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:47:28.084760       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:50:10.663969       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:51:17.774591       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:54:17.087557       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:56:01.666489       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 18:58:05.776825       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:02:51.090353       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:04:32.669112       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:06:54.778876       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:09:53.670880       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:10:50.092458       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:14:00.781953       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:18:20.673290       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:18:25.094576       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:21:44.784422       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:26:21.674970       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:28:21.096383       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:30:58.786715       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:32:40.676951       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:35:21.099196       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:36:27.788856       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:40:42.679800       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:42:42.101662       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:45:36.791289       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:46:07.682510       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:51:48.794793       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:52:32.103598       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:52:33.684594       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:57:15.797377       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:59:21.105997       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 19:59:47.686654       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:07:06.800696       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:07:34.689615       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:08:09.107989       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:13:57.803003       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:14:43.110336       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:16:53.692151       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:20:28.113928       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:21:04.805155       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:24:24.694367       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:25:58.116588       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:28:27.808049       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:33:07.696667       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:35:56.119352       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:36:30.810405       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:41:48.699318       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:41:51.121428       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:45:53.812656       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:47:45.702995       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:48:01.123463       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:52:31.814884       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:54:44.705500       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:56:29.125998       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 20:59:50.707571       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:00:00.816872       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:02:59.128184       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:05:14.819125       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:09:03.710037       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:09:39.131455       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:11:21.821687       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:18:22.712671       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:19:25.133185       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:21:09.823967       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:26:32.715722       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:27:04.136101       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:28:38.826712       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:34:10.138703       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:35:21.718244       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:36:43.828727       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:41:59.831102       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:42:30.719993       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:43:32.141867       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:47:51.833613       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:47:58.722472       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:50:26.143858       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:56:04.836004       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:57:31.723959       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 21:59:32.146231       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:04:38.838837       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:06:43.149031       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:07:21.726757       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:09:59.841453       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:15:19.151487       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:15:32.730271       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:19:14.844637       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:21:12.732744       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:24:27.154301       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:24:40.846686       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:27:35.734689       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:30:41.156788       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:33:25.737109       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:34:25.849098       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:39:05.159454       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:39:38.739649       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:40:07.851178       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:44:47.741629       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:48:37.163197       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:49:59.852881       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:52:24.743759       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:56:54.166102       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:56:58.855584       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 22:58:06.746536       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:02:53.857600       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:06:10.748833       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:06:16.168597       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:12:47.860054       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:15:48.171327       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:16:02.751851       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:18:34.862330       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:22:32.754734       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:24:03.173384       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:28:02.864395       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:29:35.757653       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:33:34.175352       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:34:58.866951       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:36:40.760168       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:40:55.869693       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:42:46.762898       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:43:27.177300       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:49:44.872112       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:50:30.765038       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:50:32.179466       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:55:13.874304       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:57:36.181105       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0429 23:59:51.767246       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:02:43.876242       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:04:29.183756       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:05:24.769735       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:11:05.878731       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:14:06.186999       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:15:19.771577       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:18:35.881163       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:20:40.773396       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:22:31.189194       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:27:09.883968       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:29:40.775864       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:30:32.191989       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:32:24.886190       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:35:45.778703       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:39:36.194213       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:40:52.888400       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:45:14.781084       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:47:40.890209       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:49:05.196883       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:51:24.783650       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:55:34.892822       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 00:58:19.199689       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:01:13.785842       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:02:09.894942       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:06:45.202761       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:09:55.787990       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:10:11.897462       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:11:53.205460       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:17:21.790143       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:18:15.899624       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:19:57.207407       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:24:44.792545       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:25:19.901806       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:28:54.210439       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:32:52.794898       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:33:16.904356       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:35:06.212358       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:39:30.906247       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:40:25.214516       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:41:55.797305       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:46:21.908959       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:46:29.216564       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:51:06.799580       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:52:33.911585       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:53:21.218415       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:57:51.913982       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 01:59:32.801617       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:00:29.221354       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:04:52.804356       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:04:57.915973       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:06:04.223641       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:10:14.807179       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:14:05.225854       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:14:19.918190       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:18:26.809752       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:19:14.228415       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:20:33.920110       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:24:06.812822       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:25:04.230343       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:29:50.922253       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:32:51.232697       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:33:55.815490       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:38:38.924480       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:39:49.235432       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:43:30.818095       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:46:18.237976       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:47:22.926729       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:51:59.240538       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:53:18.820556       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 02:54:59.929190       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:00:53.243611       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:02:51.823630       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:04:56.931396       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:07:09.245865       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:11:34.825730       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:12:20.248516       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:13:00.934177       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:16:52.828800       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:18:58.936063       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:19:01.250840       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:23:10.830947       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:24:42.939283       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:28:50.253281       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:30:33.833480       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:34:35.942066       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:35:20.255378       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:37:19.836198       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:41:10.944504       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:42:31.838047       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:44:34.257233       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:47:45.947765       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:51:03.259264       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:52:15.840793       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 03:55:30.950519       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:00:13.843946       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:00:30.262092       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:03:45.953109       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:05:33.846596       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:06:07.264078       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:09:22.955288       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:11:33.849689       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:15:10.265992       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:15:59.958043       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:17:59.852743       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:23:59.268852       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:24:45.855496       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:24:56.960232       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:29:39.271223       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:30:10.858526       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:30:50.962811       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:35:28.273452       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:38:24.860804       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:39:58.965592       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:43:32.275732       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:48:17.863772       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:48:39.278265       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:49:52.968366       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:53:58.866787       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:56:13.281288       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 04:56:17.970515       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:00:28.869284       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:01:32.972511       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:06:07.284735       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:08:32.872169       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:11:22.974892       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:11:51.287603       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:16:41.874801       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:18:13.977036       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:21:20.289479       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:24:21.979971       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:26:00.877653       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:27:59.292294       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:33:01.981799       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:33:03.295082       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:33:20.880043       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:39:28.297849       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:41:53.983662       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:42:17.882259       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:47:51.300665       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:49:21.884305       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:51:10.985966       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:52:53.303510       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 05:57:26.887070       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:00:00.988673       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:00:05.305837       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:05:13.888994       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:07:14.990975       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:08:55.307629       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:13:26.993655       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:14:29.891885       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:17:45.310256       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:18:58.996118       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:23:09.893937       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:24:17.312509       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:26:51.998511       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:30:18.314762       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:32:20.000688       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:32:56.896420       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:37:31.317185       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:37:52.003580       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:39:29.898805       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:45:09.320072       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:47:18.006544       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:47:24.901183       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:53:32.322802       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:55:28.009037       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:56:49.903316       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 06:59:50.325453       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:02:52.905321       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:03:59.011117       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:07:25.328262       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:10:35.907763       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:11:42.012968       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:16:59.331144       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:18:35.910659       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:19:03.015165       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:23:25.333660       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:25:02.017756       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:28:04.912673       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:31:01.336380       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:31:42.020141       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:37:28.338444       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:37:31.914763       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:39:42.022897       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:43:00.917269       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:43:47.341043       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:47:04.024997       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:49:28.343767       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:51:33.920360       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:53:24.027244       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:56:09.346131       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 07:57:16.922443       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:02:22.030531       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:05:19.348482       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:06:40.924875       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:12:12.033069       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:13:30.927619       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:13:33.350781       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:18:10.035589       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:18:30.930388       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:22:31.353083       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:24:59.038888       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:25:52.932364       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:29:41.355557       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:34:25.041436       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:34:37.934704       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:39:05.357775       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:42:17.044581       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:44:23.937623       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:45:08.359563       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:48:15.046726       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:50:08.940512       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:53:59.361898       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:55:49.050043       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 08:56:55.942897       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:01:33.364892       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:03:59.945469       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:05:25.053191       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:09:27.367360       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:13:58.947503       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:14:20.055285       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:14:42.369759       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:20:07.949948       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:21:04.372044       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:21:31.057609       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:26:41.059826       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:28:26.375082       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:28:41.952342       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:33:38.377452       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:33:39.061885       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:34:38.954721       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:39:00.063647       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:41:13.380313       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:41:26.957061       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:46:41.066151       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:48:12.959553       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:50:50.382729       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:54:51.068796       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 09:56:56.961748       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:00:08.384721       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:03:45.964602       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:04:08.071340       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:06:39.386749       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:10:51.966480       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:12:15.388875       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:12:50.073385       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:18:52.969525       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:18:56.390807       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:19:16.075306       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:24:08.393360       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:26:19.078365       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:28:16.971476       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:29:15.395790       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:31:58.080964       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:34:54.398258       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:36:35.973261       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:40:23.083264       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:41:42.975498       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:43:54.400539       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:46:49.977468       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:50:08.085857       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:50:35.403342       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:54:37.979839       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:55:36.405657       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 10:56:58.088685       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:00:55.408373       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:01:55.982382       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:02:50.090647       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:07:16.985419       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:09:31.093140       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:09:58.410524       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:14:48.987947       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:15:49.413593       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:16:42.095243       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:21:47.991569       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:21:50.097584       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:25:46.415623       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:27:32.994326       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:31:20.099813       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:33:27.417860       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:36:05.996725       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:38:28.419761       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:39:17.103078       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:44:06.999560       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:44:33.422429       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:47:06.105540       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:52:08.002892       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:53:34.425047       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:54:45.107254       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 11:58:32.005827       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:00:34.427372       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:04:03.109704       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:04:51.008573       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:08:12.430265       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:09:49.112257       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:10:00.010975       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:15:11.115847       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:16:11.432373       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:19:09.014110       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:21:44.118229       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:22:43.434870       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:27:15.016808       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:31:18.120675       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:32:27.437885       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:36:28.019195       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:36:57.122464       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:39:12.442620       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:43:22.021418       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:45:53.125380       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:46:22.444269       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:51:39.128077       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:52:48.024122       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:54:09.446826       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:57:14.130864       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 12:59:57.026800       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:00:31.449289       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:06:50.133882       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:07:46.451247       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:07:56.029033       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:15:01.136859       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:15:20.031088       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:15:46.453542       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:22:03.033191       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:22:09.456289       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:22:55.139588       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:28:24.458278       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:30:33.142047       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:30:41.035049       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:34:18.461139       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:37:08.144476       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:39:47.037068       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:41:22.462980       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:43:40.147308       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:49:24.039599       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:50:21.464669       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:52:54.149512       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:55:26.042224       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 13:58:56.467185       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:02:16.151400       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:04:30.044445       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:08:09.469933       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:09:10.153735       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:10:29.046616       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:14:38.155933       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:18:03.471790       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:19:24.048770       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:21:02.158698       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:24:21.474763       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:28:41.160833       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:29:18.051129       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:29:43.476514       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:37:35.054219       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:38:14.163734       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:38:31.479306       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:45:43.166451       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:46:02.056744       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:47:53.481725       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:51:13.059578       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:55:08.168650       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:55:21.484207       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 14:56:13.062265       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:04:09.486654       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:04:49.064501       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:04:54.170510       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:10:54.489210       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:11:10.172253       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:14:45.066530       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:19:54.491349       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:19:56.174272       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:23:53.068980       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:25:16.493487       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:27:20.176214       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:30:21.071129       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:32:55.496525       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:35:38.073875       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:36:19.178724       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:40:30.498440       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:43:53.076876       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:45:36.180544       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:48:42.500244       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:49:56.078867       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:54:36.183772       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:55:22.502341       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:59:47.186648       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 15:59:55.080863       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:04:05.504969       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:05:39.082722       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:07:27.189004       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:12:42.507089       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:13:28.085577       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:14:30.192123       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:17:53.510031       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:21:24.088570       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:23:34.511940       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:24:00.193958       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:29:08.197037       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:30:03.090606       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:32:30.514331       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:35:41.199739       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:39:18.517120       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:39:26.092466       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:44:55.201730       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:45:52.094512       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:48:24.520235       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:52:18.096626       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:52:58.203981       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:54:28.522870       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:57:38.099568       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 16:58:31.206218       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:00:30.525109       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:07:24.208855       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:07:26.101998       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:07:52.527570       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:13:54.211198       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:15:52.103498       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:17:46.530043       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:23:42.214164       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:25:39.106300       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:27:21.532168       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:29:31.216946       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:34:27.108957       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:35:53.534270       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:39:09.219870       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:42:37.112013       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:44:13.536603       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:45:39.222048       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:51:09.224758       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:51:57.539003       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:51:59.114463       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:57:22.116746       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:57:42.541548       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 17:59:02.227131       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:03:00.544862       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:03:10.119239       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:07:24.229118       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:10:18.121816       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:12:33.547466       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:15:59.232173       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:16:09.124291       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:21:55.549695       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:22:48.234293       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:23:25.125879       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:27:20.551561       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:29:22.237405       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:32:05.128016       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:33:44.554052       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:36:40.239956       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:39:29.131111       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:41:30.556433       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:45:06.241764       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:46:26.133687       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:47:42.558829       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:54:08.136580       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:54:19.244121       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 18:54:54.561872       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:01:50.246390       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:03:31.138761       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:03:40.563888       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:08:40.565502       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:09:17.248961       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:10:22.141122       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:16:27.251745       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:17:06.568450       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:19:02.143859       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:22:15.254213       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:22:55.570262       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:26:44.146400       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:27:29.256207       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:28:45.572522       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:32:34.259161       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:34:21.149031       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:35:44.575102       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:38:08.261454       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:40:18.151779       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:43:53.264216       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:44:00.577056       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:45:21.153744       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:49:32.579563       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:50:01.266237       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:55:14.156818       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:58:10.582300       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 19:58:57.268332       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:02:30.159177       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:04:53.585246       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:04:59.270608       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:09:36.161752       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:12:06.272996       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:13:25.587335       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:18:31.589192       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:19:00.164792       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:20:31.275100       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:23:59.591734       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:24:26.167346       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:27:40.277472       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:29:11.594728       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:32:02.170096       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:34:04.279270       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:36:02.597128       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:41:31.172813       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:42:54.281887       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:45:27.599313       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:49:11.285118       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:49:49.174553       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:51:51.601439       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:55:05.287997       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:55:21.176399       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 20:56:59.603243       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:03:09.605451       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:04:42.290114       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:04:54.178529       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:10:42.608983       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:10:55.180953       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:11:16.293382       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:16:37.611484       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:18:45.183364       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:19:14.295675       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:23:02.614357       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:25:17.185403       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:28:52.298050       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:29:58.617265       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:30:46.188462       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:37:20.300582       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:38:29.620208       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:38:34.191497       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:44:06.623190       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:45:39.193366       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:47:00.302655       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:50:36.625893       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:52:06.305708       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:54:04.195605       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:57:34.628085       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 21:58:03.308024       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:01:10.197831       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:04:33.310884       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:04:39.630739       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:09:28.200534       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:10:28.632884       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:13:19.313316       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:15:56.202900       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:19:30.635653       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:20:25.316119       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:21:51.205847       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:25:22.637608       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:26:30.318770       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:31:41.640207       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:31:43.208794       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:32:32.321042       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:37:27.643509       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:37:50.211252       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:42:11.323030       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:43:11.646450       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:44:19.213743       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:48:17.650147       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:50:16.325557       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:53:33.216560       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:56:37.328676       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:58:15.653133       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 22:59:34.219236       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:03:44.331092       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:06:50.221518       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:07:24.655656       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:12:18.224168       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:13:25.333671       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:14:39.658268       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:18:23.226138       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:23:21.335978       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:23:22.660226       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:27:34.228805       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:29:53.338269       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:30:27.661793       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:32:57.231610       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:36:19.340969       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:38:14.233422       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:38:50.663745       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:43:05.342702       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:46:47.665847       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:47:38.236338       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:52:44.344877       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:52:55.668847       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:55:07.238608       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0430 23:59:43.671183       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:00:06.347133       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:01:13.240963       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:04:48.673663       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:09:07.349816       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:10:49.242994       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:11:45.676055       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:14:17.352127       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:17:10.678965       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:19:58.245666       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:23:07.354229       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:25:38.681525       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:29:04.248406       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:32:43.357302       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:34:07.683520       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:36:22.251294       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:41:01.685690       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:41:10.359483       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:41:58.254267       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:46:29.361030       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:47:50.687711       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:51:36.256496       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:53:39.690352       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:55:50.363157       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 00:59:42.692668       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:00:03.258869       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:02:59.365644       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:06:37.261718       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:08:01.368250       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:09:19.695235       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:14:20.698380       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:14:58.264485       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:17:43.370974       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:22:32.701162       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:22:33.267035       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:24:25.374453       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:28:05.269249       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:30:15.377122       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:32:21.704246       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:34:14.271627       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:40:03.379251       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:42:18.706966       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:43:28.273493       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:48:52.381892       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:50:20.275485       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:50:46.709375       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:55:50.278376       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 01:58:31.384832       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:00:17.711757       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:03:52.281507       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:07:39.386784       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:08:17.715012       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:09:38.284262       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:13:46.716949       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:15:54.389533       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:19:13.286844       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:22:12.391740       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:23:01.719357       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:25:08.289474       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:29:55.394293       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:29:58.722196       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:30:55.291399       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:35:44.397405       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:35:45.724021       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:39:34.293386       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:40:48.400280       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:44:12.725996       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:46:24.295941       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:49:09.403147       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:49:24.728223       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:54:23.298848       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:55:20.730445       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 02:58:40.405438       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:00:00.301142       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:03:49.733016       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:06:02.304126       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:06:59.407436       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:09:46.735029       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:13:44.306597       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:16:32.410547       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:16:59.737344       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:20:07.309399       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:23:33.738975       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:25:00.412867       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:26:26.312422       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:31:35.415038       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:32:15.315103       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:32:25.742295       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:37:57.744392       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:39:36.417882       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:41:10.317202       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:45:40.747503       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:47:43.420266       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:49:20.319935       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:54:15.423256       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:54:36.749736       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 03:55:57.322981       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:01:04.326067       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:01:43.426150       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:03:04.752472       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:09:29.329446       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:10:52.428521       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:12:39.755188       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:17:37.430802       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:18:13.331473       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:18:45.757751       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:26:04.760273       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:26:25.333625       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:27:03.433457       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:32:19.436488       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:32:52.762956       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:33:26.336227       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:39:32.338714       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:40:04.765474       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:41:48.438498       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:46:28.768287       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:47:21.341551       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:48:09.440191       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:52:20.770759       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:53:27.443039       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 04:56:57.343441       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:00:39.773896       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:02:20.445440       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:03:00.345237       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:09:21.776989       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:11:36.448581       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:12:21.347366       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:15:59.779879       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:17:40.450895       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:21:33.349616       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:22:51.781765       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:24:22.453884       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:30:29.352591       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:30:51.783975       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:31:55.455794       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:36:03.354606       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:37:47.458376       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:40:40.786573       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:43:48.461035       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:44:38.356817       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:48:28.788609       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:51:25.463432       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:53:32.359577       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:56:30.791282       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 05:58:42.361575       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:01:17.465800       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:04:58.794219       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:06:42.364467       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:08:28.468131       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:14:30.797075       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:15:46.366989       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:16:00.469945       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:21:31.799689       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:21:36.368963       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:23:01.471853       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:27:31.802208       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:30:58.474452       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:31:22.370859       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:33:51.804557       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:36:10.476838       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:39:08.807753       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:40:21.373956       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:43:06.478801       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:44:19.809696       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:47:47.376030       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:49:32.812874       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:50:16.480666       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:54:16.379167       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:55:10.815318       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 06:57:37.482526       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:02:28.382197       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:04:46.817441       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:05:38.484586       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:09:31.385300       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:10:11.820313       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:14:30.486479       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:17:13.388255       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:19:32.822862       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:20:13.488346       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:25:25.825813       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:26:11.390462       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:27:09.491227       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:31:11.393140       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:31:33.827863       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:33:36.493571       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:39:11.496890       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:40:32.395333       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:41:04.830147       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:45:56.500923       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:46:35.397879       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:48:19.832596       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:53:59.503389       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:54:41.399532       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 07:54:53.834294       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:02:17.402618       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:02:20.836763       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:02:23.506153       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:07:51.405576       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:08:33.508296       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:09:27.838788       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:15:02.510548       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:16:29.841295       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:17:33.407714       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:23:16.409558       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:23:49.513443       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:24:55.844320       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:32:56.411737       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:33:03.515942       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:33:44.846683       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:38:19.414523       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:39:13.849294       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:42:40.517658       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:44:22.416465       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:45:15.851505       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:48:31.520239       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:53:07.853794       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:53:22.418519       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:57:49.523281       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 08:59:18.855594       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:01:43.420553       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:05:03.858054       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:07:32.526251       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:09:28.422551       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:11:24.860619       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:14:42.528871       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:16:45.862788       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:18:55.424725       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:20:32.530954       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:22:52.865187       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:25:10.427630       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:26:05.533592       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:30:16.867587       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:30:21.430189       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:35:36.432998       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:35:43.536341       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:37:48.869950       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:41:29.538612       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:43:02.435233       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:44:07.872412       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:47:43.541829       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:48:26.437434       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:52:12.874350       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:55:37.439395       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:57:15.544970       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 09:58:33.877039       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:02:58.443112       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:04:38.878984       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:05:27.547590       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:08:03.445323       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:10:59.880991       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:15:18.549477       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:16:05.448174       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:20:40.884030       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:22:42.449895       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:25:09.551992       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:25:53.886289       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:29:41.453234       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:31:07.553659       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:31:29.888627       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:36:20.455609       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:36:46.891099       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:37:38.556408       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:42:36.458855       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:44:12.558343       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:45:17.893228       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:49:35.461348       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:50:30.560515       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:52:24.895272       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:59:02.897994       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:59:19.463689       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 10:59:35.562482       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:07:26.901058       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:08:44.465741       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:09:29.565392       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:14:59.903486       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:16:19.567732       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:16:35.467883       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:22:33.470423       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:23:01.570137       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:23:19.905870       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:28:44.572203       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:31:21.472993       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:32:56.907930       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:38:34.574831       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:38:44.910344       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:39:57.475024       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:46:28.913642       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:46:42.576974       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:48:47.477827       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:51:58.579599       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:52:02.916606       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:57:03.918835       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:57:10.480210       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 11:59:11.582399       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:04:28.482294       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:04:42.920682       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:06:30.584620       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:10:57.922402       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:12:52.484954       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:14:25.587767       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:20:50.924465       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:22:34.487633       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:22:34.590240       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:26:33.926330       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:29:17.489470       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:32:10.592119       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:32:28.928412       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:34:53.491959       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:39:26.931974       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:40:05.594812       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:43:08.494283       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:45:05.934767       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:49:04.596756       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:49:45.496897       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:55:01.937960       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:58:01.599599       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 12:58:35.498847       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:01:06.940733       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:04:15.501443       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:06:21.943208       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:06:26.602626       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:12:29.605171       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:13:51.504051       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:15:07.944897       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:19:01.506268       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:20:38.607100       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:21:19.946867       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:25:24.509344       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:28:06.949616       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:29:47.609359       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:31:23.511252       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:33:12.951890       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:37:39.611701       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:38:18.954200       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:41:15.514106       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:45:10.614371       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:48:03.957078       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:48:06.516433       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:50:23.616647       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:55:15.518274       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:57:47.959996       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 13:59:47.618860       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:02:53.521003       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:05:25.962186       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:05:52.620620       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:09:15.523770       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:10:38.965326       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:15:16.622841       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:17:43.526402       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:18:53.967497       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:20:27.624723       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:24:53.970307       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:27:16.528558       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:28:04.626880       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:33:02.972600       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:33:55.629260       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:34:24.531380       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:40:29.631693       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:40:33.533876       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:41:42.975344       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:46:47.977976       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:46:49.633668       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:48:16.535691       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:54:37.980803       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:54:56.635669       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 14:55:58.537977       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:01:11.638343       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:03:09.983311       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:03:34.539808       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:08:15.641082       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:11:39.542251       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:12:37.986014       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:16:50.642948       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:19:40.544644       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:22:03.989299       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:24:26.644711       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:28:14.546564       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:30:18.647936       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:31:10.992396       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:36:53.654236       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:37:35.548764       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:39:21.995174       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:44:41.656617       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:47:22.997788       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:47:29.551002       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:49:53.658984       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:55:22.661999       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:56:38.552801       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 15:56:51.000066       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:01:43.663945       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:01:53.555234       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:03:02.002244       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:07:43.557064       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:08:01.666138       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:09:08.004889       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:14:44.008184       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:15:15.669452       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:15:56.559186       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:21:01.672336       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:21:54.010695       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:22:59.561478       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:29:50.674646       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:30:19.563228       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:31:07.012439       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:36:19.015565       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:38:25.565184       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:39:26.677046       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:45:51.017522       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:47:19.567370       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:47:55.678934       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:51:04.020436       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:54:18.569721       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 16:55:37.681325       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:00:45.023189       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:00:56.683245       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:01:19.571735       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:08:33.685812       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:08:51.026184       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:09:26.573727       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:17:21.688165       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:17:27.027972       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:18:36.576536       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:24:38.030410       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:24:57.690452       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:26:23.578183       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:33:30.032810       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:33:37.693245       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:34:29.580549       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:41:42.035629       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:42:10.582636       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:42:57.695782       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:47:41.038497       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:51:32.697448       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:51:37.585190       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:53:31.041342       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:59:15.587896       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 17:59:55.699759       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:02:14.043424       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:05:28.702336       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:07:46.590965       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:10:45.045946       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:14:33.705644       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:14:53.593623       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:17:45.048217       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:21:50.708468       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:21:56.595340       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:26:38.051013       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:27:48.711446       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:30:21.597097       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:32:13.053612       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:33:15.714093       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:36:47.599625       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:40:40.055494       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:41:06.716522       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:44:37.601818       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:48:06.058402       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:49:40.718886       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:53:39.604428       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:56:09.060779       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 18:56:12.720876       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:00:21.607349       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:01:34.722976       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:05:06.062921       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:05:47.609840       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:10:51.725698       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:13:12.612152       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:13:22.065658       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:19:03.067816       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:19:43.728444       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:22:36.614964       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:24:08.070660       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:27:46.731069       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:31:16.617793       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:32:09.072645       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:33:17.734037       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:40:09.620107       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:40:39.075171       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:42:46.736463       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:45:38.622099       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:48:47.738626       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:49:15.077967       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:51:19.624892       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:56:22.741579       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:57:01.627585       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 19:57:04.080161       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:02:52.629493       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:03:48.744420       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:05:25.082024       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:09:24.746835       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:10:51.084653       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:12:01.631584       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:16:34.749420       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:17:41.633477       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:20:50.087005       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:24:57.635688       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:26:23.751669       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:30:23.637824       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:30:41.089631       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:33:14.753766       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:36:03.639868       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:39:11.091957       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:41:12.756145       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:43:57.642468       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:48:08.094459       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:50:01.758301       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:53:31.646061       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:56:09.760652       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:56:42.096468       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 20:59:42.648198       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:03:27.762835       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:05:01.099255       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:09:29.650382       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:10:09.101022       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:10:57.765525       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:15:55.103688       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:17:33.768073       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:18:36.653143       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:22:56.770441       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:25:50.106069       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:26:57.655621       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:30:50.772803       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:31:27.108556       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:35:51.657748       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:36:27.110425       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:40:29.775596       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:41:55.659866       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:42:18.112449       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:47:07.661736       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:48:00.778151       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:51:33.114818       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:55:32.779975       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:56:48.664299       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 21:59:44.118049       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:03:30.782475       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:04:25.667144       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:08:18.119968       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:11:22.669082       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:11:35.784586       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:15:54.122636       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:17:13.786610       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:17:41.671411       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:22:53.124758       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:24:47.674070       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:25:56.789384       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:28:33.127599       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:31:49.792225       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:34:36.677240       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:36:39.130009       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:41:37.794249       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:44:23.679203       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:46:05.132139       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:46:53.796937       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:51:32.682365       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:51:58.798958       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:55:10.134372       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 22:58:53.801626       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:00:10.685078       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:01:58.136868       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:04:52.803982       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:08:26.139456       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:09:55.686980       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:12:44.810201       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:17:06.141766       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:17:52.689824       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:18:58.812451       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:24:32.144288       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:25:10.692004       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:28:07.814758       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:33:20.146800       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:33:42.817108       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:35:09.693969       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:39:03.149805       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:41:42.819625       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:43:55.696274       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:46:19.152702       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:51:16.698103       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:51:25.822426       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:55:09.155318       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:56:34.700438       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0501 23:58:18.824168       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:00:41.158223       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:04:42.703342       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:05:40.826541       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:06:09.160292       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:10:45.705720       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:12:41.163293       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:13:07.829098       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:16:08.708792       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:20:10.165988       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:20:35.831163       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:22:27.710730       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:26:32.168221       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:28:48.713361       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:30:08.835133       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:33:10.170946       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:34:14.715424       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:39:10.838036       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:39:57.718207       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:40:21.172868       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:47:29.721260       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:48:04.174883       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:48:44.840452       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:52:53.723399       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:53:12.177220       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:57:42.843418       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 00:59:47.725280       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:00:02.179323       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:02:50.846336       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:06:56.728197       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:07:56.181781       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:08:25.848509       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:15:14.730504       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:15:36.850693       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:16:58.183700       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:21:52.732690       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:24:19.185884       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:25:01.853024       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:30:02.734645       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:31:50.856077       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:32:37.188225       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:37:07.737370       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:39:50.190527       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:40:56.858579       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:44:52.192824       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:45:37.740654       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:49:29.861352       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:51:36.742749       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:53:37.195371       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:56:52.864461       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 01:57:51.745267       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:00:17.197668       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:05:59.200507       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:06:37.867456       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:07:37.748017       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:13:51.869710       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:15:36.203027       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:16:04.750883       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:19:37.872571       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:21:01.205086       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:25:46.753484       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:28:18.206892       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:28:31.875578       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:31:29.755862       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:34:30.878467       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:36:36.208949       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:37:52.758778       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:43:43.881005       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:44:54.211690       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:45:03.760637       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:50:39.883431       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:53:09.762589       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:53:21.213781       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:58:10.764964       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 02:58:29.886260       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:00:22.216611       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:07:20.889187       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:07:28.766823       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:09:52.219470       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:12:40.769338       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:15:18.891442       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:19:32.221638       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:21:17.771872       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:23:51.894198       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:25:13.224598       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:29:27.773726       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:29:29.896820       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:32:32.227719       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:37:32.776384       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:39:25.898587       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:41:23.230665       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:46:46.779405       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:47:38.900436       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:48:14.232935       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:53:31.903021       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:53:56.235066       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 03:54:04.782356       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:01:33.905794       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:02:26.237792       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:03:24.784682       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:08:32.240322       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:08:53.908983       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:11:11.787600       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:16:22.243435       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:18:02.910988       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:20:02.789320       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:26:14.247096       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:26:58.791204       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:27:52.913005       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:33:05.249221       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:33:57.915375       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:34:55.793547       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:39:58.917740       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:40:00.795843       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:41:28.252262       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:46:09.920223       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:46:51.797545       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:49:32.253845       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:55:41.922451       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:56:14.799277       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 04:58:11.256280       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:04:08.801833       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:04:26.924868       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:07:37.258430       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:09:51.804096       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:12:26.926937       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:16:12.260990       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:17:59.806396       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:18:41.929380       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:22:26.263261       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:24:24.931871       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:27:48.809269       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:28:24.265203       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:31:33.934515       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:35:57.267200       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:36:10.811515       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:39:33.937084       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:43:06.270399       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:45:22.814190       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:49:02.939283       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:52:42.272121       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:54:15.816191       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 05:55:15.943238       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:00:09.818503       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:00:47.274556       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:02:48.945218       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:05:47.821443       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:08:43.277774       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:09:09.948276       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:12:14.823408       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:18:31.281022       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:18:52.951008       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:19:33.826021       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:27:14.284182       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:27:14.828119       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:28:32.952855       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:33:45.286742       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:36:36.830595       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:36:58.954726       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:40:48.289721       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:45:18.957286       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:46:12.833259       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:47:30.292521       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:50:30.959827       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:53:06.295103       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:56:04.835858       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:59:32.297574       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 06:59:48.962361       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:01:42.838304       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:05:34.300514       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:09:06.964496       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:10:30.841054       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:13:54.303873       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:17:51.967442       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:18:39.843406       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:23:10.306215       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:25:58.845436       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:26:49.970022       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:30:45.308540       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:31:45.847544       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:33:02.973039       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:38:08.311970       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:38:26.974900       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:40:44.850058       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:46:05.314328       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:46:41.976989       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:46:53.852844       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:52:02.855561       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:54:06.979518       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:55:49.316357       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 07:58:47.858187       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:00:08.982094       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:01:18.319215       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:06:03.984668       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:06:26.321696       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:06:47.860887       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:15:01.987303       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:16:05.324290       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:16:41.863681       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:21:42.326885       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:25:00.989855       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:26:13.865868       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:26:52.328968       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:31:47.992690       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:32:52.867822       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:36:50.332060       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:37:12.994897       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:42:42.870189       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:44:22.334844       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:47:08.997332       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:50:53.872290       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:52:13.337937       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:53:06.000110       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:57:14.875633       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:57:17.339673       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 08:58:51.002417       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:04:12.341962       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:05:32.004599       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:06:21.878971       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:13:05.007753       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:13:26.343674       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:16:15.881306       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:20:53.009956       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:23:00.346066       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:23:55.883487       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:30:06.885781       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:30:42.011936       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:30:58.348997       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:37:08.888935       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:37:35.350982       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:38:27.014360       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:43:03.891372       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:43:30.017081       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:43:42.352983       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:49:15.019142       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:49:56.893301       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:53:07.355817       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:56:40.021838       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:57:38.896241       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 09:58:08.357771       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:01:40.024524       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:05:56.360204       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:06:53.898262       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:10:10.026354       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:12:09.900254       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:13:59.362580       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:17:22.029074       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:19:45.902635       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:20:59.365396       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:23:09.031079       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:28:49.905605       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:29:27.367112       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:33:01.033370       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:34:49.907818       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:35:44.368983       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:40:58.910318       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:42:36.035329       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:43:16.371273       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:48:52.373542       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:50:08.913273       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:51:00.037697       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:55:07.375954       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:55:29.915366       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 10:57:05.040110       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:01:33.917817       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:02:24.042101       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:03:05.378315       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:07:25.920627       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:11:21.044445       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:12:35.380133       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:13:12.923372       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:18:35.382553       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:19:12.925550       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:19:57.047409       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:26:18.928300       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:26:40.049721       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:26:49.385115       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:33:34.387443       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:34:23.930602       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:35:57.051611       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:39:05.389458       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:42:40.933164       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:44:22.054381       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:46:18.391557       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:50:05.056086       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:51:26.935970       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:51:43.393418       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:56:09.058591       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 11:58:39.938552       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:00:33.395578       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:01:45.061085       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:04:06.941606       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:07:08.397964       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:08:50.063404       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:11:45.944088       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:16:31.400382       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:16:55.946076       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:18:46.065964       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:23:14.403025       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:25:19.949203       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:26:18.069037       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:31:17.951998       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:32:49.405049       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:33:06.071218       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:39:51.407382       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:40:00.954575       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:40:42.074787       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:47:13.409999       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:47:42.956135       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:48:16.077659       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:54:46.412360       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:56:24.958231       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 12:56:50.080350       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:01:43.414642       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:01:47.960557       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:05:00.082394       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:07:12.962351       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:07:31.417269       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:11:06.084823       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:13:40.419838       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:16:39.086749       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:17:09.965051       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:22:14.422387       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:22:42.088988       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:26:15.967304       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:27:50.424701       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:32:20.092023       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:34:48.426753       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:34:53.969763       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:39:59.971660       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:41:15.429750       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:41:59.094557       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:45:46.974174       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:48:53.096532       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:50:06.432272       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:52:43.976267       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:55:09.099589       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 13:58:40.434714       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:00:48.101997       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:00:49.978765       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:06:03.103694       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:06:20.437468       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:10:35.981233       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:11:54.439409       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:15:07.105734       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:20:22.109078       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:20:26.983319       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:21:01.443285       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:29:31.445510       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:29:53.985281       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:30:00.110876       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:35:49.447662       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:36:38.987755       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:38:02.112873       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:44:35.990212       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:44:37.450085       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:45:37.115074       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:52:41.452969       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:53:37.992977       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:55:03.117570       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:58:34.454865       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 14:59:17.995022       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:01:43.119592       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:05:02.456879       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:05:11.997764       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:11:12.459709       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:11:14.122228       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:11:39.999583       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:17:57.002315       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:18:15.462227       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:19:36.124312       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:25:28.464284       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:26:52.126564       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:27:12.004944       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:32:25.466495       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:35:15.008425       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:35:49.128778       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:39:21.469616       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:43:38.010722       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:44:26.130761       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:47:28.472027       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:49:47.013487       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:53:25.132995       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:56:18.474693       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 15:56:51.015579       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:00:24.135274       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:02:43.017483       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:03:44.476719       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:08:47.137924       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:09:04.019392       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:12:53.479203       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:15:42.139708       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:18:13.021372       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:18:22.481482       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:24:28.024394       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:24:36.141959       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:28:19.483261       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:32:05.144087       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:32:59.027188       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:33:19.485724       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:37:58.147251       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:39:26.487534       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:40:56.029244       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:45:18.150039       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:47:06.490033       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:48:51.031175       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:53:02.491695       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:53:30.152906       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 16:58:06.034370       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:02:19.155713       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:02:38.493673       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:04:09.036570       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:09:19.158488       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:11:45.495658       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:13:02.039138       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:18:42.498174       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:18:56.161029       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:22:50.041276       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:25:02.500495       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:25:28.163395       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:32:30.044014       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:32:39.164955       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:34:56.502842       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:38:56.046221       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:41:50.166725       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:42:22.505185       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:47:18.048830       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:50:10.168787       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:50:48.507941       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:56:52.051551       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:57:53.171154       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 17:59:05.510052       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:04:33.512457       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:06:09.053974       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:07:38.173494       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:11:54.055825       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:13:39.176161       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:14:14.514399       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:19:54.058534       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:20:28.178019       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:20:41.516429       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:26:40.180552       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:26:46.060939       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:30:22.519065       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:36:07.182454       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:36:45.063102       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:39:21.521027       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:43:54.184728       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:45:30.066199       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:46:42.523469       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:49:19.187158       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:51:05.069192       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:56:20.525469       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 18:59:08.190522       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:00:57.072143       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:03:30.527704       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:05:56.192881       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:09:25.074569       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:11:58.530308       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:14:51.195015       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:16:26.077181       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:19:12.533285       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:19:53.197189       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:24:49.079115       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:26:56.536033       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:28:46.199411       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:31:35.081797       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:34:15.201657       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:35:33.538737       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:40:26.084554       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:42:26.204211       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:45:03.540630       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:47:46.207132       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:49:49.086828       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:54:51.543193       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:56:40.088734       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 19:57:26.209372       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:01:10.545295       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:05:24.091315       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:06:54.547703       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:07:21.212210       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:11:07.093876       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:13:02.549739       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:15:47.214689       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:20:53.096646       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:21:39.216568       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:21:58.552366       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:27:15.555172       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:29:37.098729       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:31:14.219124       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:33:30.557622       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:36:30.100527       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:37:34.222071       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:43:06.559600       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:43:30.103100       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:44:24.223980       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:48:55.561718       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:52:25.226455       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:52:32.105397       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:57:39.564009       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 20:58:09.228377       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:00:28.107052       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:03:58.230982       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:06:01.566975       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:06:32.109375       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:09:02.233275       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:15:10.569114       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:15:24.111572       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:18:02.235833       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:24:11.571586       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:24:23.113693       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:26:18.238361       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:31:18.240286       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:32:33.574223       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:32:37.116249       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:37:54.119954       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:37:59.242883       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:42:22.576546       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:47:43.245739       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:47:49.122471       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:49:47.578779       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:56:03.581712       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:56:55.124386       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 21:57:09.247805       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:05:27.584193       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:05:43.126467       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:07:02.249989       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:12:05.129752       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:13:10.252526       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:14:39.586946       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:18:56.255788       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:20:05.132387       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:21:57.589369       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:27:14.258675       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:27:57.591824       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:29:06.134910       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:33:30.260768       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:35:59.594844       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:37:08.137762       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:40:14.263288       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:43:39.140341       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:44:28.597534       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:49:12.266546       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:50:06.599157       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:51:12.142437       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:56:47.601142       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:58:24.269068       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 22:59:56.145115       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:02:21.603287       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:05:07.147092       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:07:24.271976       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:09:13.606097       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:10:49.149095       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:15:32.274586       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:16:13.608305       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:19:38.151784       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:23:36.277560       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:24:19.610584       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:25:10.154274       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:30:11.612729       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:31:50.280446       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:34:09.157079       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:36:55.614859       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:37:08.282404       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:41:50.158892       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:43:54.285344       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:46:36.617640       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:47:24.161061       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:50:43.287337       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:51:40.619575       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:52:57.163228       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:56:17.290615       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0502 23:59:01.621618       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:02:03.166040       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:04:56.292576       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:08:19.168668       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:08:59.624182       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:13:51.295070       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:16:59.627236       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:17:50.170981       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:21:10.298096       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:23:05.173107       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:26:26.629148       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:28:18.300114       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:29:21.175718       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:33:12.632145       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:36:29.302297       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:38:09.177808       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:39:51.634357       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:41:30.304489       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:44:57.180385       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:45:22.636838       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:49:15.306685       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:51:22.183155       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:53:58.638799       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 00:55:42.309888       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:00:57.185536       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:02:51.641029       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:02:53.312286       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:08:12.314816       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:09:32.187828       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:12:42.643285       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:15:42.190238       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:16:21.317348       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:20:44.646599       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:22:01.192856       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:23:48.319432       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:30:36.649867       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:31:19.321292       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:31:30.195682       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:36:06.651981       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:37:02.197587       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:40:09.324201       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:42:38.200142       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:43:41.664941       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:49:32.326626       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:51:46.667383       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:51:51.203109       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:58:28.669297       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:59:11.204679       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 01:59:18.328846       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:04:44.330994       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:05:13.671371       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:05:19.207147       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:13:42.674158       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:14:02.333810       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:15:18.209177       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:20:21.336052       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:20:53.211368       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:22:58.676183       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:28:31.339199       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:29:27.213315       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:30:52.678457       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:34:40.215247       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:34:51.341615       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:37:19.680596       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:41:47.343821       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:42:02.217098       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:43:18.682994       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:48:09.346660       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:48:29.218896       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:49:59.685759       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:55:11.688982       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:55:44.220852       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 02:57:15.349496       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:03:14.224162       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:04:14.691218       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:07:11.351847       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:11:19.227309       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:14:07.694235       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:16:08.354803       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:21:10.229675       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:21:22.695938       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:23:59.357629       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:26:32.698488       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:27:24.232525       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:30:42.359955       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:34:39.234860       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:35:05.700720       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:40:03.361889       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:41:16.702385       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:42:12.237126       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:46:06.363809       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:50:18.705395       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:51:08.366422       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:51:41.239672       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:56:36.707175       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 03:58:30.242609       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:00:22.369140       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:03:39.708840       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:06:47.245851       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:09:37.371392       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:12:00.248814       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:12:37.710799       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:18:35.373405       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:19:09.712783       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:20:13.251655       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:26:18.715151       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:26:41.254629       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:26:52.375104       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:32:43.717747       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:32:46.256787       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:34:07.377215       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:39:12.380125       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:39:13.719303       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:42:28.258865       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:45:33.721415       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:48:22.382837       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:50:39.261816       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:54:26.724194       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 04:55:08.385541       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:00:11.264376       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:02:52.388388       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:03:35.726793       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:07:18.266981       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:08:56.728844       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:10:53.391733       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:13:54.270370       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:16:16.731629       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:17:07.394708       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:22:27.274665       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:23:37.396984       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:24:39.733429       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:29:13.277211       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:29:43.399414       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:34:25.736293       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:35:00.279499       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:35:32.402090       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:41:15.738688       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:42:19.404741       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:43:31.282168       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:48:55.407743       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:49:18.741297       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:52:00.284312       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:56:52.410342       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:57:30.286777       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 05:57:50.743331       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:05:31.414075       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:05:44.289775       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:06:58.745335       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:11:03.416585       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:12:53.747393       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:15:42.292559       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:17:57.418891       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:21:17.294896       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:22:27.749755       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:25:58.421789       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:28:17.297670       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:30:33.751551       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:35:51.423092       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:36:42.299658       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:37:54.754285       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:41:48.425775       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:43:16.302261       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:46:41.757162       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:48:32.305158       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:51:28.428565       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:52:46.760360       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 06:54:19.307831       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:00:27.762631       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:01:08.310441       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:01:13.430551       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:06:20.432449       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:08:53.312628       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:09:38.765105       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:12:08.434713       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:14:55.767270       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:16:50.315550       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:17:45.437630       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:21:54.769165       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:22:40.317779       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:27:38.440287       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:28:32.771821       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:30:01.319708       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:36:12.774415       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:36:16.443088       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:37:57.321737       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:41:32.777211       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:42:45.444543       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:46:32.324555       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:47:06.779526       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:49:42.447076       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:55:16.327161       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:56:42.782749       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 07:57:50.449641       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:02:41.329331       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:04:16.785068       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:05:26.452422       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:08:50.332044       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:12:58.787544       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:14:09.453943       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:18:02.789585       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:18:47.334279       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:23:28.456242       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:24:24.792198       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:25:42.336774       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:31:33.338835       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:31:54.459260       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:33:18.794400       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:38:28.341081       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:40:41.461888       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:41:23.796387       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:47:14.342931       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:48:06.463829       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:48:24.799081       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:55:42.466255       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:56:23.345074       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 08:58:15.801129       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:03:41.469492       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:05:04.347178       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:06:29.803952       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:11:30.472155       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:13:11.349845       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:13:45.806469       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:17:53.474869       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:19:48.352227       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:21:52.809367       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:26:53.477005       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:28:22.355341       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:31:38.811727       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:35:11.478662       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:36:36.358239       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:37:53.814563       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:41:35.481294       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:43:53.360350       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:46:06.816970       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:48:40.484101       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:52:56.363131       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:53:31.819080       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:55:39.486687       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 09:59:33.365111       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:01:08.821942       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:03:25.488847       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:06:37.824204       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:07:09.367242       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:11:12.490637       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:12:29.369105       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:14:10.826237       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:20:16.371537       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:21:08.492818       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:22:18.828244       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:26:40.494746       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:27:42.374203       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:29:29.830286       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:34:19.376195       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:35:24.497048       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:35:43.833110       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:40:51.378438       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:43:45.498709       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:45:35.835836       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:49:36.381676       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:53:02.837532       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:53:05.500650       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 10:55:50.384191       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:00:54.840510       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:01:37.386085       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:01:48.503138       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:06:55.842959       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:10:50.505418       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:10:52.388198       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:12:34.844958       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:16:06.507374       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:18:37.389869       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:18:49.847188       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:23:33.510273       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:27:47.849366       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:27:59.391654       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:32:00.512516       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:33:08.851423       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:37:22.394150       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:39:35.515268       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:39:54.853949       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:44:33.397142       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:44:50.517506       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:45:37.856794       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:53:49.407425       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:54:38.519083       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 11:55:32.858696       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:00:38.861340       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:01:19.410768       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:01:51.521774       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:08:30.524796       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:10:24.863409       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:10:44.413006       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:13:30.527015       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:17:01.415841       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:19:25.865861       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:21:34.529374       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:25:07.417834       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:25:19.868644       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:29:14.532166       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:30:55.871297       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:34:26.420138       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:36:06.534778       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:37:42.873318       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:39:47.422402       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:43:04.875732       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:43:14.536753       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:48:20.538944       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:49:21.424647       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:51:15.877935       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:54:48.429576       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 12:56:15.541074       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:00:56.881097       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:02:02.542933       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:02:43.431555       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:08:37.434031       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:10:05.545193       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:10:15.883888       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:16:50.886315       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:18:23.547486       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:18:24.436894       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:24:31.439079       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:24:37.888160       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:25:59.550280       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:30:11.890614       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:30:27.443866       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:34:48.552213       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:38:06.893434       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:39:06.445683       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:42:04.555324       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:45:47.448398       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:47:12.896024       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:49:44.557436       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:50:53.450531       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:52:47.898609       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:56:08.453375       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:57:38.559555       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 13:58:55.900955       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:04:29.562003       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:05:24.455061       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:07:55.903726       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:11:29.563955       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:12:15.457185       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:17:54.906497       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:19:20.566348       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:20:52.459243       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:23:20.909165       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:28:54.912039       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:28:57.568464       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:30:28.462167       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:34:56.914849       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:37:08.570657       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:37:09.464257       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:40:04.917069       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:44:17.466625       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:45:20.573491       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:46:01.918844       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:52:29.921054       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:53:17.468959       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 14:53:22.575803       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:01:48.471065       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:01:52.924116       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:02:37.577453       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:07:12.473382       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:08:33.926027       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:12:20.580124       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:14:52.475220       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:15:29.928041       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:20:08.477522       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:21:06.582707       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:24:49.930654       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:27:33.585233       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:28:09.479948       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:30:57.932773       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:35:42.482641       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:36:19.588480       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:37:14.935315       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:42:03.590614       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:43:57.937840       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:44:49.484258       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:47:03.593088       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:50:00.940366       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:52:19.487136       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:53:36.595041       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:58:06.489163       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 15:59:08.942859       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:00:02.597422       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:06:26.492019       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:07:05.600003       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:07:47.945118       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:13:38.602584       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:14:51.946999       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:15:54.494798       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:20:54.949445       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:22:10.605093       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:22:13.497161       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:29:07.607995       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:29:55.951577       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:32:09.498842       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:35:41.954071       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:36:02.610775       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:40:05.501818       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:42:00.613217       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:45:33.956841       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:46:22.504348       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:50:03.615572       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:52:34.506983       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:53:41.959589       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 16:59:51.618434       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:02:21.961739       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:02:30.509270       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:07:07.620223       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:09:23.964282       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:12:20.511257       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:12:25.621778       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:18:55.967398       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:19:36.624449       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:22:00.513294       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:24:12.969778       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:27:34.515428       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:28:08.626689       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:33:25.517972       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:34:00.972497       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:35:31.628863       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:38:47.519718       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:42:06.630727       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:43:52.975563       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:44:11.521281       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:49:20.633514       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:51:06.978199       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:53:39.523944       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:57:10.636538       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 17:58:17.980317       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:00:28.526140       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:04:38.982387       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:05:42.528210       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:06:44.639737       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:11:06.984592       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:13:47.641666       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:14:49.530838       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:16:10.986856       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:21:48.644590       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:22:48.533082       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:23:06.989460       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:27:38.647515       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:29:18.991387       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:32:28.535796       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:33:10.650041       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:37:56.537658       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:39:12.993546       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:39:30.652452       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:44:35.540040       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:45:35.995759       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:48:19.655025       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:53:28.543153       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:53:33.997452       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:55:39.658002       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 18:58:34.546633       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:02:41.000621       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:04:45.660757       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:06:22.549109       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:08:03.003147       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:12:51.663232       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:14:55.551825       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:16:55.005369       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:21:18.666433       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:22:10.008194       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:23:30.554169       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:30:11.669361       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:31:37.010228       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:33:14.556840       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:39:46.671293       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:40:35.012686       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:42:18.559629       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:46:43.015418       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:47:58.673322       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:48:17.562111       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:54:37.018108       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:56:10.565214       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 19:56:15.675638       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:01:09.020053       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:01:15.568132       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:01:45.677811       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:08:08.680600       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:08:37.022453       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:10:11.570642       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:13:51.025044       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:13:51.682737       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:16:37.573433       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:20:32.027273       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:20:44.684805       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:22:50.575561       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:29:46.031665       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:29:54.686662       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:30:05.577422       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:35:28.033587       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:37:59.580507       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:38:24.689015       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:44:35.035693       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:45:06.582763       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:47:34.691164       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:50:15.585418       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:52:52.693068       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:54:24.037664       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 20:58:54.588393       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:01:36.695767       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:03:49.040139       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:08:00.698241       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:08:30.590425       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:12:08.043111       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:13:27.701110       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:16:04.593336       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:19:13.045539       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:21:21.704248       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:22:07.596079       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:26:32.706667       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:28:42.047644       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:30:27.598598       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:34:29.708708       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:38:34.049660       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:40:09.601653       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:41:25.711107       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:45:40.603841       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:46:45.052255       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:48:23.713530       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:55:15.055057       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:55:17.605705       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 21:56:46.716020       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:02:27.608840       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:03:20.718080       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:05:14.057550       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:08:05.612083       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:11:57.059281       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:12:12.719836       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:13:26.614399       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:17:06.061447       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:20:33.616966       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:21:38.721985       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:22:08.063533       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:28:39.619592       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:30:04.065627       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:31:16.723586       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:35:30.621770       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:36:22.725521       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:36:39.068014       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:43:42.728046       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:44:16.069766       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:45:08.624594       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:52:41.730676       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:53:14.626984       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:53:23.072158       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 22:58:28.073790       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:00:09.733233       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:01:11.629276       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:06:14.736253       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:06:55.631685       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:06:59.075823       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:13:55.077557       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:15:04.634285       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:16:04.738458       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:22:10.080378       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:22:35.740847       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:23:45.637332       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:28:53.083278       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:31:01.743191       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:32:36.639532       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:36:01.745500       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:37:10.085512       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:40:38.641702       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:44:06.088413       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:45:47.748348       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:48:41.644046       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:53:21.090117       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:53:25.751051       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0503 23:54:33.645789       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:00:43.753677       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:00:55.648264       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:01:52.092608       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:07:12.756156       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:08:40.095084       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:08:52.650672       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:14:04.759093       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:16:13.097488       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:16:15.652480       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:20:17.761157       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:21:15.100272       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:22:53.654075       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:26:46.101950       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:27:43.764086       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:28:04.656215       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:35:43.658426       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:35:43.766665       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:36:04.103627       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:41:07.661307       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:45:23.768679       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:45:55.105579       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:47:27.663341       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:53:06.771257       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:53:42.107625       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:56:09.666121       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 00:59:31.773615       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:01:49.110220       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:04:34.668295       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:04:56.775608       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:07:07.112338       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:10:23.777561       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:12:46.115257       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:14:24.670287       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:18:54.780442       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:21:23.673095       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:21:57.117783       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:25:01.782809       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:30:34.120929       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:31:09.675277       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:32:08.784615       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:37:48.123833       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:39:37.787720       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:40:06.677685       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:43:32.126300       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:46:26.791031       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:48:27.679686       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:49:39.128291       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:53:32.682690       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:53:32.793737       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:59:18.131285       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 01:59:54.795265       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:00:02.685555       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:08:30.798430       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:08:56.133857       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:09:42.687493       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:14:54.136035       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:15:05.803036       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:18:20.690312       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:20:33.138482       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:23:48.692467       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:24:34.804955       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:29:34.695037       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:29:39.807108       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:29:55.140383       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:37:10.142650       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:37:26.808848       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:38:05.698647       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:43:17.700685       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:45:42.810559       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:46:52.145326       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:52:32.703222       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:54:51.147361       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 02:55:10.812869       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:01:12.150012       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:02:25.705580       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:03:04.815444       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:10:33.708293       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:10:53.152621       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:12:21.817415       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:15:57.711434       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:19:48.155821       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:21:26.820135       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:21:31.713396       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:26:45.157951       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:27:14.822602       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:29:58.716385       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:33:16.160473       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:35:00.825088       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:39:41.719257       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:41:38.163437       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:43:38.827492       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:46:52.721127       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:49:11.829324       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:50:09.166657       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:54:39.831804       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:55:31.168750       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 03:56:36.723983       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:03:04.726614       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:03:19.834350       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:03:24.171034       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:09:04.173982       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:10:51.836329       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:11:09.729371       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:15:38.176721       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:16:06.838841       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:20:27.732094       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:21:22.178588       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:23:59.841108       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:28:14.181327       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:29:01.735670       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:30:42.843686       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:35:53.845801       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:37:23.183949       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:38:24.738318       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:42:15.848465       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:46:43.186251       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:47:11.740568       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:49:46.850768       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:52:21.743198       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:56:00.189438       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 04:58:46.853497       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:00:17.745783       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:04:07.856214       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:05:02.192218       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:07:59.748959       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:09:11.858345       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:12:37.194967       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:17:14.751286       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:17:46.860592       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:20:06.197126       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:23:11.862801       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:23:40.754024       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:25:28.199865       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:30:14.865017       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:30:53.755767       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:34:46.201816       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:38:50.867370       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:39:39.758412       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:42:46.204542       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:45:02.870246       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:46:38.761074       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:48:38.207024       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:51:24.872549       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:52:19.763971       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 05:58:10.209396       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:00:38.875170       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:02:07.766123       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:04:40.212304       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:05:38.877116       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:10:28.214615       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:10:47.769070       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:12:58.879432       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:17:06.216584       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:19:32.771590       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:20:45.882494       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:25:16.219153       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:26:28.773322       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:26:28.885259       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:31:22.221617       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:33:41.886944       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:34:42.775154       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:38:49.889713       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:39:58.223426       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:41:13.777161       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:46:12.225791       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:46:36.892172       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:47:42.779359       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:54:55.228625       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:55:22.894239       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 06:55:45.781051       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:01:02.783272       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:01:08.230425       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:03:14.896455       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:09:16.786053       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:09:17.232410       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:11:01.898938       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:16:30.788245       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:18:01.234479       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:20:41.901268       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:22:43.790626       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:24:56.236897       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:25:51.904312       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:31:17.792801       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:32:39.906349       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:34:22.239586       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:38:04.909176       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:38:18.794693       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:43:53.242396       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:44:59.796237       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:47:17.911429       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:50:18.245127       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:52:42.913483       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:52:43.798624       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:56:59.248218       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 07:58:43.918254       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:00:25.800734       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:02:09.250499       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:04:25.920377       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:07:21.802678       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:07:39.252727       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:12:24.805805       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:13:22.255516       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:13:31.922856       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:18:33.258627       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:19:13.807651       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:22:27.924493       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:25:43.810686       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:26:40.260414       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:27:43.926964       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:31:01.812772       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:32:13.263229       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:35:51.929298       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:36:57.815398       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:38:51.265952       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:41:51.932558       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:43:43.817625       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:44:38.268005       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:49:18.935272       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:49:51.819609       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:52:22.274603       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:54:28.937361       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:58:08.276977       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:59:11.821669       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 08:59:46.939677       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:04:13.279190       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:06:38.824102       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:08:00.942154       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:11:50.826792       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:13:27.283055       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:15:19.944419       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:19:24.829078       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:22:34.285357       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:24:10.946886       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:25:31.831813       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:29:14.288284       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:32:22.949360       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:32:56.834477       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:37:57.952393       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:38:33.291117       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:41:10.836712       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:43:19.954851       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:44:57.293617       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:49:55.838690       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:50:55.957390       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:52:44.296461       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:55:35.841009       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 09:56:25.959571       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:00:07.298631       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:01:15.842955       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:03:11.961522       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:08:59.300556       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:09:53.845300       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:12:40.964479       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:16:06.303217       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:17:16.847318       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:18:10.967273       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:23:56.969821       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:24:50.305737       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:26:58.849364       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:29:21.972133       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:31:01.308462       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:34:10.851812       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:38:20.974617       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:39:55.310789       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:42:49.854137       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:46:25.977174       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:47:41.312855       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:49:14.856037       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:52:58.979507       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:55:52.316029       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 10:57:02.858863       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:00:57.981535       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:03:34.318604       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:05:27.861151       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:10:07.983438       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:12:11.321459       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:14:37.863275       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:15:09.986129       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:17:46.324330       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:21:31.988533       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:24:33.865459       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:27:37.326626       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:29:55.991283       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:31:46.868391       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:33:17.329076       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:35:26.993591       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:39:49.330858       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:40:40.870277       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:43:46.996784       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:46:26.871933       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:48:56.333886       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:51:58.999423       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:54:39.335818       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:54:45.874264       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 11:57:55.001416       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:01:14.875888       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:02:39.338593       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:06:19.003107       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:08:55.341858       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:09:41.878284       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:12:48.005784       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:14:47.344026       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:18:54.880686       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:19:35.008571       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:22:24.346208       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:24:17.883892       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:27:03.010564       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:29:20.886878       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:30:09.348261       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:32:27.012965       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:36:11.888856       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:38:06.014811       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:38:44.349784       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:42:51.890530       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:45:01.017971       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:47:01.351803       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:52:46.892801       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:52:50.354329       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:54:07.020769       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 12:58:24.356314       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:01:18.894726       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:01:58.023181       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:04:49.359436       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:06:33.896948       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:10:02.026122       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:12:49.362334       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:15:24.899088       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:16:36.028653       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:18:15.365056       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:21:59.030651       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:24:11.901054       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:24:45.367642       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:28:00.033227       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:32:17.903208       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:33:32.369793       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:33:35.035408       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:38:40.905485       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:39:15.371958       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:40:13.037560       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:44:04.907440       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:45:19.374344       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:47:18.040167       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:52:36.377150       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:52:54.909744       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 13:53:47.042318       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:00:06.912390       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:00:33.380317       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:00:48.044440       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:06:11.914614       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:07:44.382836       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:08:26.046696       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:13:09.917115       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:14:31.384635       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:17:07.048980       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:22:19.919138       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:23:58.387129       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:25:19.051620       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:29:17.389336       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:31:19.921350       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:32:47.054010       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:36:38.923521       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:36:55.391650       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:38:54.056072       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:44:53.394193       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:46:12.925732       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:47:31.058449       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:51:38.397141       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:55:38.060508       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:55:47.928015       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 14:58:56.401264       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:01:03.063114       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:01:22.930485       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:05:50.403868       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:08:05.932753       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:09:45.065550       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:12:53.406829       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:14:22.935292       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:19:31.068336       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:19:50.409444       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:20:51.937648       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:25:26.070842       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:26:21.940000       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:29:36.411426       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:32:23.073798       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:32:37.942360       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:38:15.413954       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:39:22.076216       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:41:57.944689       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:44:37.078822       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:46:16.415781       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:48:14.947363       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:51:41.081330       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:55:18.949495       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:55:59.417591       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 15:59:31.083802       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:01:40.419938       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:03:18.951304       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:06:32.086182       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:11:18.422644       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:12:33.953324       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:15:07.089137       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:17:42.955109       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:20:24.425177       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:21:49.091084       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:24:01.957152       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:26:31.428453       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:31:47.093493       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:33:21.959269       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:35:57.430664       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:37:05.095643       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:40:56.961560       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:43:26.433078       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:44:28.098340       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:48:19.963628       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:51:24.435563       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:51:57.100968       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:55:09.965842       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 16:56:55.437499       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:00:10.967894       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:00:29.102865       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:02:55.439821       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:08:15.105371       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:09:17.970509       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:12:07.441963       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:14:33.972424       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:14:51.107520       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:18:11.444467       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:22:04.974270       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:22:55.109773       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:25:31.446837       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:27:31.976735       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:31:03.111856       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:32:14.448703       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:36:03.113811       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:36:40.979588       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:38:56.451122       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:43:38.116546       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:43:57.981786       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:44:41.453350       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:52:49.119289       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:53:07.983952       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:53:26.455898       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 17:58:24.986673       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:01:52.121114       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:02:05.458535       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:07:50.989288       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:08:36.460345       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:11:36.123001       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:14:36.992478       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:16:20.462126       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:17:46.124826       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:23:12.995765       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:23:53.127069       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:26:05.463814       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:30:02.998063       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:30:42.130039       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:34:49.466884       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:36:20.132478       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:37:08.000460       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:40:07.468776       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:42:45.002464       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:43:22.135054       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:45:11.471142       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:48:21.004440       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:50:47.473787       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:51:47.137714       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:57:26.007522       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:59:28.139817       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 18:59:37.476219       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:03:02.010512       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:05:01.142335       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:07:34.477848       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:10:46.012889       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:13:57.145495       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:16:39.480378       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:17:06.015360       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:23:51.482552       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:23:52.147994       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:26:54.017803       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:30:15.485123       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:31:28.150351       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:36:41.020865       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:36:58.152631       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:38:51.487303       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:43:12.023260       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:43:39.155066       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:48:07.490541       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:49:19.157072       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:50:25.025818       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:56:02.492868       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:57:33.029061       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 19:57:53.159313       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:04:36.030787       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:05:30.494723       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:06:56.161642       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:11:06.496730       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:11:54.033364       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:12:17.163633       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:16:13.498952       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:21:05.035302       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:21:23.166558       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:24:00.501506       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:28:25.168361       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:29:27.037201       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:31:39.503896       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:36:53.039334       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:37:26.170009       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:39:51.506365       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:42:29.041304       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:46:49.172283       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:48:48.508661       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:49:50.043247       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:53:49.174579       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:55:53.045422       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 20:56:17.511139       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:02:36.047434       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:03:28.176462       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:03:53.513062       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:07:39.050289       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:09:02.179302       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:13:35.515346       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:14:50.052733       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:15:09.181748       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:18:42.518076       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:23:14.184672       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:24:22.055421       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:27:36.520637       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:29:22.186945       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:32:14.058621       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:35:20.522472       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:37:21.191225       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:37:41.061299       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:41:20.524375       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:42:29.193809       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:47:08.064574       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:47:44.196373       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:49:00.526989       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:53:39.067084       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:54:05.198656       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:54:11.529662       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 21:59:40.069231       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:01:36.200417       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:03:08.531513       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:06:57.071984       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:10:27.202942       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:12:21.533554       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:16:32.074620       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:17:10.205078       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:22:05.536221       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:22:06.076351       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:22:32.207150       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:27:40.538883       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:28:05.209642       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:31:41.078074       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:34:32.541817       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:35:35.211340       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:41:38.080854       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:43:23.213750       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:43:26.544075       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:48:32.545807       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:50:02.216569       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:51:01.083214       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:56:48.548328       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:59:01.218639       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 22:59:38.086246       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:05:07.550313       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:05:12.088557       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:06:26.220369       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:13:40.553339       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:13:52.090988       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:13:54.222728       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:19:03.093519       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:20:15.224655       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:22:54.555170       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:25:50.096520       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:29:04.226756       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:29:39.557121       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:31:29.099679       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:35:29.559322       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:38:54.229031       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:39:51.102283       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:44:07.565768       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:48:14.104102       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:48:24.231770       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:51:15.568952       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:53:28.106212       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:55:17.233885       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0504 23:58:56.108980       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:01:02.571058       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:02:45.236529       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:06:05.573623       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:07:52.111077       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:11:43.239172       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:11:47.576521       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:15:06.113152       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:19:52.241798       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:20:50.115069       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:20:52.578584       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:26:13.244721       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:27:07.580026       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:27:41.117238       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:34:02.246603       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:34:54.582230       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:36:11.119625       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:43:47.249557       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:44:07.584384       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:45:19.122017       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:49:13.587331       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:50:30.252264       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:51:24.124777       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:54:42.590318       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:56:46.126996       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 00:57:51.254723       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:00:15.593003       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:02:56.129442       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:05:18.256877       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:09:15.595438       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:09:34.132206       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:12:21.259959       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:15:24.134362       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:17:57.597664       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:18:03.261558       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:21:00.136633       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:24:24.263448       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:24:34.599977       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:26:47.138645       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:31:30.265624       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:34:26.601835       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:36:11.141607       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:37:36.268251       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:40:29.604819       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:44:44.143944       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:45:09.270538       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:50:03.607592       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:50:18.146010       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:51:26.272477       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:56:23.149110       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 01:56:23.610157       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:00:47.275345       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:05:14.612447       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:05:25.151236       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:09:38.277392       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:10:42.153499       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:10:43.614383       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:15:06.280188       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:16:45.155796       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:19:33.616390       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:20:51.282738       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:25:02.158089       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:26:16.618405       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:29:09.285424       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:33:47.620732       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:34:03.161058       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:38:24.288286       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:40:14.164153       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:40:47.623413       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:46:01.290763       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:47:40.626349       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:47:59.166737       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:53:33.293294       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:55:54.629201       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:57:51.169483       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 02:59:41.295315       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:01:13.631531       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:03:59.172062       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:08:07.634187       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:08:45.297499       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:10:24.174590       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:14:10.299860       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:17:02.636489       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:19:32.302763       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:19:48.176965       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:26:14.179810       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:26:54.638993       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:28:34.305718       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:33:04.641577       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:34:06.307982       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:35:18.182215       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:39:30.310394       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:40:20.644214       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:45:04.184023       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:46:57.646187       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:49:19.313211       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:53:00.186388       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:53:39.648600       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:58:37.189525       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 03:59:04.315611       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:02:44.651244       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:07:22.192703       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:08:03.318046       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:10:08.653784       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:15:13.194579       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:16:54.656613       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:17:18.320662       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:20:34.197263       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:22:52.659094       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:27:00.322621       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:27:53.661417       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:30:11.199743       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:33:35.325447       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:33:47.663291       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:35:30.202399       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:39:49.665607       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:42:02.328047       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:42:24.205035       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:45:01.667832       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:49:37.330198       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:50:48.207134       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:53:34.669973       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:57:06.209744       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 04:59:19.332454       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:02:53.673455       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:03:55.212299       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:05:41.334741       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:12:18.214417       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:12:23.675758       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:15:26.337520       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:20:43.216094       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:22:18.677407       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:24:29.340493       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:28:25.218977       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:29:56.679620       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:31:35.342650       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:34:13.221587       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:39:35.682205       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:39:41.344594       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:40:41.224215       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:45:00.346490       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:47:11.684484       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:48:22.226747       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:54:23.348599       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:54:29.687061       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:56:51.229351       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 05:59:48.351301       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:00:56.688862       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:02:13.231715       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:09:00.234223       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:09:21.691118       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:09:41.353868       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:16:06.693696       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:16:12.236104       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:17:27.356511       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:25:16.237845       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:25:59.695869       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:26:52.358548       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:32:09.361230       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:34:49.240074       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:35:01.697740       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:40:43.363891       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:43:02.699798       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:43:18.243139       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:48:02.367702       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:50:08.702734       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:53:17.246187       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:55:12.705276       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:57:42.370554       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 06:59:59.248759       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:02:26.708171       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:05:58.251582       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:06:58.372604       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:07:49.710488       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:13:19.254292       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:13:46.712449       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:16:40.375583       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:19:26.715255       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:23:12.257129       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:23:32.378190       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:25:02.717574       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:30:15.380677       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:32:48.259179       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:32:58.720475       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:37:58.383780       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:38:05.722072       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:42:36.261458       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:44:11.385866       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:44:31.724002       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:47:58.264188       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:50:15.388760       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:53:21.725743       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:55:24.266850       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:58:31.728593       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 07:58:59.390997       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:04:54.269267       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:06:45.731323       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:08:16.393323       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:10:08.271830       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:12:11.733446       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:15:32.273789       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:17:25.395271       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:21:27.735387       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:22:57.276318       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:23:18.397147       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:29:22.737100       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:30:09.278269       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:30:16.398820       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:36:26.281516       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:36:57.400703       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:38:58.739864       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:41:59.284116       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:44:37.403383       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:46:19.742749       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:50:18.286704       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:51:07.406479       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:52:49.745585       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 08:56:09.409360       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:00:12.289136       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:00:39.747586       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:04:11.412141       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:08:07.292341       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:10:35.750189       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:11:54.414104       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:16:18.294684       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:17:02.752527       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:20:29.415961       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:23:16.754739       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:23:18.296676       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:27:31.418527       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:29:42.757346       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:32:24.298825       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:36:26.421398       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:37:19.759375       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:41:53.300842       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:42:16.424263       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:45:47.762341       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:47:37.427000       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:47:55.303272       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:53:34.429144       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:53:39.305811       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 09:55:30.765177       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:01:12.307805       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:01:22.431128       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:03:28.767435       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:08:22.434388       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:10:32.310534       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:10:51.769694       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:16:13.437019       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:16:23.771601       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:20:31.314004       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:25:49.438948       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:26:12.773506       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:28:48.316754       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:34:33.776179       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:35:31.443104       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:35:39.319462       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:42:20.322121       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:43:28.778210       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:45:24.445446       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:49:56.781101       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:50:59.324998       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:52:05.447980       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:56:52.327367       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:59:25.782984       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 10:59:55.450159       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:03:20.330032       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:06:43.785487       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:09:22.452894       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:11:43.787787       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:13:01.333065       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:17:47.790369       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:18:07.454758       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:18:20.335297       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:25:29.337383       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:25:29.456302       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:25:29.792526       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:30:47.340370       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:32:22.458686       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:33:04.793990       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:36:58.342673       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:40:09.460998       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:41:41.796190       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:45:43.463136       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:46:13.345663       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:48:12.798199       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:52:33.348136       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:53:57.465296       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 11:55:02.800962       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:01:13.350228       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:01:43.803739       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:03:16.467084       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:07:10.352883       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:09:53.805946       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:11:03.468761       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:12:23.355098       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:16:49.470957       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:18:02.808578       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:22:18.357628       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:22:30.473394       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:23:05.810672       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:27:59.475688       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:28:22.813380       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:31:19.360001       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:35:48.816195       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:37:10.478092       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:40:55.362368       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:43:58.818834       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:44:09.480675       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:45:57.364690       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:52:43.483160       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:52:50.820753       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:54:02.366751       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:58:59.823131       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 12:59:13.369438       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:00:47.485736       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:04:44.825712       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:06:51.488292       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:08:18.371109       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:10:39.827579       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:13:44.490170       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:14:16.373060       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:18:33.830076       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:21:30.376162       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:21:42.492252       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:25:26.832397       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:29:24.494464       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:31:20.378050       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:31:54.835180       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:36:57.497286       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:40:50.837448       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:40:52.380364       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:46:00.499747       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:46:08.382102       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:46:32.839266       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:52:15.384165       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:54:56.502114       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 13:55:57.841055       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:00:46.386875       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:02:02.503722       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:04:51.842857       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:07:56.505365       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:09:06.389656       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:13:37.507404       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:14:03.845371       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:18:13.392135       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:21:04.848279       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:22:30.509485       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:28:06.394505       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:28:52.850285       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:30:46.511412       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:33:41.396848       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:35:00.852887       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:40:42.513523       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:42:25.398949       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:44:53.855232       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:47:17.516583       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:48:14.401714       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:51:29.857671       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:53:52.403758       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:56:12.518605       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:57:45.860201       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 14:59:44.406531       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:03:47.521511       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:05:28.862317       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:06:07.409556       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:09:40.523712       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:11:25.411889       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:12:04.864173       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:18:15.414746       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:19:29.526246       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:21:27.866804       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:24:12.417383       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:27:55.529447       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:28:52.869322       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:31:55.419227       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:36:16.872224       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:36:54.531659       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:40:16.421780       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:41:54.874858       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:45:12.534016       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:45:39.423309       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:50:43.877141       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:53:19.536686       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:55:08.425685       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 15:58:11.880524       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:00:44.539295       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:04:32.428167       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:07:48.883364       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:08:07.542003       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:13:27.431404       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:14:14.544016       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:14:54.885205       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:20:10.434627       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:23:20.546712       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:23:37.887209       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:28:17.437450       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:31:27.549770       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:32:29.889203       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:35:05.439681       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:39:47.892109       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:40:24.552210       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:41:53.444083       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:47:52.446136       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:48:30.894492       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:50:09.554887       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:53:56.447810       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:54:57.896325       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:55:26.556890       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 16:59:39.450515       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:00:37.559731       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:03:58.898646       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:06:14.562125       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:08:56.453605       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:13:08.900814       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:15:29.564646       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:16:23.456280       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:20:33.903979       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:23:16.568042       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:25:54.458821       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:30:22.906545       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:32:47.571052       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:35:41.461124       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:36:50.908515       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:38:08.573429       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:44:05.463720       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:45:22.575422       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:45:56.911458       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:50:33.577456       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:51:44.465563       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:53:39.914167       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:58:07.467862       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 17:58:23.580249       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:00:17.916822       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:04:24.582107       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:05:42.470452       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:07:58.919168       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:11:24.583727       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:13:52.472555       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:16:44.588774       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:17:01.921760       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:23:37.474720       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:23:37.591261       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:27:00.924022       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:30:58.477225       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:31:05.593233       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:33:46.926072       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:40:20.595811       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:40:22.478996       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:41:59.928636       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:45:36.598551       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:48:18.930393       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:49:32.480691       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:52:17.600542       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:56:13.482586       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 18:56:24.932760       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:00:48.603501       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:01:55.934312       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:04:19.484718       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:09:43.605952       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:10:25.487012       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:11:42.936326       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:16:22.489695       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:18:37.608770       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:20:30.938752       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:22:20.491396       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:26:26.941620       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:27:34.611319       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:29:25.493351       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:32:38.613182       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:35:56.495815       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:36:05.944002       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:41:47.946078       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:42:05.615821       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:43:09.497863       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:49:49.948619       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:50:50.500244       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:51:35.617996       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:54:59.951087       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:57:35.503223       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 19:58:50.620291       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:02:37.505617       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:02:57.952867       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:07:05.622329       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:10:37.955483       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:11:39.507536       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:15:27.625021       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:17:52.958362       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:20:47.509963       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:21:46.627079       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:26:27.960842       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:28:38.512771       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:31:12.629367       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:35:52.963624       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:36:00.514843       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:37:22.631252       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:42:57.516951       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:43:31.966191       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:46:49.633336       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:51:30.519584       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:53:03.968535       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:53:40.636307       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:56:47.522092       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 20:59:59.638683       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:01:21.971336       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:02:52.524807       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:05:39.640880       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:09:07.527124       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:11:18.973864       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:15:10.529825       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:15:13.643858       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:18:10.976110       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:23:27.646556       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:24:20.531653       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:26:39.978727       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:28:28.649127       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:29:32.534315       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:34:32.981114       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:34:49.651621       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:37:16.536512       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:39:47.984114       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:41:30.653624       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:42:55.539845       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:46:30.986201       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:50:50.655562       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:51:10.542815       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:51:53.988307       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:56:23.545675       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 21:58:30.657451       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:00:26.991126       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:04:33.548456       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:05:16.659944       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:07:42.994133       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:10:46.662173       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:10:57.550709       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:14:22.996309       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:17:15.553454       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:17:15.663986       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:20:57.999136       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:24:49.555989       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:27:05.665331       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:27:46.001848       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:31:22.557976       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:33:28.666828       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:33:40.003891       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:38:08.560429       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:40:31.669420       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:43:03.005730       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:47:52.562697       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:48:10.672478       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:50:28.007483       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:53:47.674470       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:56:55.009647       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 22:57:51.565605       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:02:24.676892       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:04:13.011677       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:06:35.567693       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:09:29.678834       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:12:37.014653       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:13:31.569587       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:18:24.680611       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:19:16.571153       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:21:37.018489       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:27:20.573438       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:27:50.683331       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:31:34.021707       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:35:52.576411       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:36:42.685805       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:41:20.024525       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:41:33.578366       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:44:52.688377       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:47:06.026771       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:50:07.580265       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:50:24.690339       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:52:32.029270       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:55:25.692616       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:58:07.031237       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0505 23:58:26.582842       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:03:11.033348       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:04:52.584677       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:05:03.694635       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:09:42.036140       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:13:10.586446       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:13:10.696363       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:16:39.039050       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:19:38.588634       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:21:55.698520       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:24:54.041360       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:27:25.591096       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:29:19.700886       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:31:03.043870       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:34:21.593716       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:35:33.703267       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:37:35.046703       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:41:43.705982       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:44:11.596204       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:44:24.048882       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:50:15.708224       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:52:11.052159       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:53:17.598932       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 00:57:09.709941       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:00:03.054797       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:01:25.601419       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:04:40.712563       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:06:17.057528       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:09:00.603811       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:11:20.060439       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:11:49.715405       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:14:18.606611       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:18:49.062685       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:20:00.717770       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:23:27.609507       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:26:42.719775       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:27:58.065133       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:30:51.611819       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:34:49.722296       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:35:04.066885       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:36:41.614519       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:40:47.068821       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:41:57.617286       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:44:39.724628       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:46:21.071164       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:49:01.619717       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:50:15.727479       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:52:04.073514       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:58:20.075708       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:58:43.730408       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 01:58:59.622471       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:03:44.732413       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:06:00.077644       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:08:35.624390       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:10:06.734300       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:13:08.079998       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:15:19.626736       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:16:09.736248       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:22:05.738461       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:22:38.082013       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:23:13.628330       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:29:45.740983       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:31:09.630358       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:32:25.084544       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:34:52.743635       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:40:05.631946       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:41:43.086450       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:42:51.746315       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:45:06.633989       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:49:58.089681       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:50:10.636311       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:51:19.749064       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:55:52.092106       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 02:58:06.638886       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:00:25.751159       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:01:02.094378       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:05:53.753486       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:07:06.096560       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:07:30.640995       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:13:26.643795       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:14:56.098813       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:15:06.755564       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:19:56.101312       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:20:16.646187       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:24:29.757542       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:29:25.103618       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:29:43.648608       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:32:17.760472       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:36:43.650658       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:38:07.106610       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:38:45.762877       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:43:35.653522       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:45:42.108879       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:46:24.765258       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:51:22.660799       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:53:04.111754       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:55:00.767537       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 03:57:10.663739       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:00:13.114689       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:04:04.770225       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:06:27.666570       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:08:43.116830       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:11:31.668824       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:12:08.772167       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:14:00.119264       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:17:50.774157       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:18:05.670720       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:23:00.121445       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:23:05.673453       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:27:10.776385       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:30:00.675379       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:31:46.123661       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:37:00.778863       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:37:15.677342       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:41:14.126364       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:42:30.679624       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:46:03.780954       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:46:38.129144       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:50:38.682183       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:53:04.783175       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:56:23.132301       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 04:58:10.683690       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:01:29.785329       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:05:09.135325       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:06:55.686302       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:08:30.787374       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:10:40.138349       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:15:22.790380       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:16:31.688843       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:19:37.140850       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:22:05.691003       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:23:24.792722       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:25:05.143693       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:28:00.693276       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:29:07.795335       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:32:10.146380       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:35:03.797415       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:35:13.695341       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:40:19.800445       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:40:39.148347       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:41:55.696867       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:45:41.802651       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:48:06.150386       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:48:43.699156       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:51:00.805304       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:55:39.701925       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:55:52.153054       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 05:58:00.807247       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:02:17.704355       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:02:33.154707       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:03:15.809450       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:09:39.812127       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:10:19.156770       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:11:14.706046       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:17:43.709019       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:18:53.814531       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:19:42.158979       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:25:25.711542       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:25:36.161806       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:27:37.817151       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:34:11.164060       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:34:30.713384       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:37:14.819700       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:40:13.715747       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:42:31.166385       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:45:59.821653       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:46:11.717865       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:51:40.720563       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:51:51.168749       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:54:55.823687       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:58:07.171033       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 06:58:44.723261       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:04:21.825898       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:05:18.173774       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:06:17.725235       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:11:24.828534       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:12:41.727233       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:12:58.175591       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:17:05.831053       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:20:41.729173       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:20:48.177891       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:22:30.833074       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:27:43.731804       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:27:52.180304       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:29:38.834757       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:33:12.182147       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:35:19.837342       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:37:32.734342       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:39:40.184367       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:40:53.840273       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:44:51.186678       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:47:30.736634       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:49:50.842899       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:50:17.188680       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:53:57.739206       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:59:33.844974       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 07:59:54.191075       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:00:06.741269       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:07:43.847353       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:08:39.743603       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:09:34.193503       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:15:38.746154       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:15:59.849663       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:17:38.196662       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:22:09.852311       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:22:56.202503       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:24:45.748431       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:28:05.205274       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:31:09.854057       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:32:15.750970       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:34:56.207608       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:37:24.856368       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:38:52.753200       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:40:22.210204       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:44:51.755660       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:45:24.213102       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:46:19.858811       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:52:10.215759       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:52:40.758071       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:54:30.861412       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:57:19.218937       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 08:59:34.760430       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:01:17.863670       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:04:17.221021       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:06:06.762242       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:08:31.865750       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:12:17.222534       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:12:55.764332       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:16:04.868613       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:20:10.224592       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:20:22.766600       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:23:00.870415       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:25:26.226409       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:27:28.769241       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:31:22.229564       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:31:44.872482       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:36:09.771590       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:40:07.874666       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:40:49.232116       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:43:26.774302       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:47:03.877117       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:47:07.234205       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:49:45.776239       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:53:48.236145       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:54:50.778495       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:56:18.879380       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 09:59:51.780209       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:02:19.881105       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:02:33.237735       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:08:13.782602       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:08:27.239568       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:08:38.883421       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:13:45.885936       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:17:03.242045       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:17:57.784828       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:22:02.888404       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:26:14.786846       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:26:56.244707       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:31:12.890989       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:31:25.788888       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:34:11.246416       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:39:57.791356       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:39:57.892812       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:40:41.248373       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:47:08.895038       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:49:13.794365       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:50:10.251166       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:54:46.897567       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:57:31.253975       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 10:58:36.796417       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:02:36.899872       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:06:25.255824       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:08:32.798198       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:11:29.902281       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:13:54.800331       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:14:39.257780       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:17:31.904678       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:20:27.259635       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:20:38.802857       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:26:01.262198       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:27:05.906811       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:28:04.805444       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:33:11.264718       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:35:00.808403       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:36:10.908966       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:38:52.266793       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:43:14.911665       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:43:32.810269       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:45:08.269480       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:52:07.272567       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:52:09.914265       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:53:25.812824       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:58:30.815889       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:58:34.274613       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 11:59:46.916527       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:04:51.818037       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:04:51.918144       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:06:16.276579       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:11:56.278736       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:12:47.820739       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:13:09.919578       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:19:24.280763       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:19:39.922233       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:20:35.822500       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:27:20.283014       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:28:18.925183       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:28:41.824686       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:33:43.285468       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:34:15.826927       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:36:48.927619       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:42:39.929660       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:42:52.828968       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:43:06.287429       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:48:07.832158       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:49:37.289437       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:50:33.932219       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:53:48.834343       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:57:49.292625       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 12:57:59.934680       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:00:47.835895       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:07:03.295202       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:07:26.937402       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:08:21.838026       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:15:03.840487       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:15:28.297670       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:16:17.940321       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:21:38.300685       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:23:10.843159       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:26:08.942418       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:29:50.845666       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:31:17.303178       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:32:26.944996       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:36:30.848281       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:37:57.306111       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:38:08.947028       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:41:42.850815       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:43:58.307975       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:44:26.949102       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:49:00.310876       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:49:07.852526       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:51:37.951710       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:54:17.313803       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 13:54:59.854605       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:00:26.857328       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:00:58.954393       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:02:18.316446       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:07:18.860304       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:07:55.957035       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:11:34.318443       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:13:56.862144       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:17:50.959737       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:19:42.864672       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:19:57.320606       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:26:20.866579       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:27:10.323020       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:27:22.961665       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:32:03.868382       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:33:31.964416       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:35:42.326049       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:40:15.966735       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:41:44.871028       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:42:04.327988       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:47:46.969336       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:47:50.872731       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:50:27.330286       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:55:39.333604       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:56:54.874495       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 14:57:09.971811       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:04:02.335711       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:04:37.876989       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:06:08.974308       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:11:05.878991       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:11:47.338205       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:14:11.976853       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:19:54.881790       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:20:41.979064       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:21:15.341153       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:26:23.884323       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:27:15.981886       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:29:37.343170       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:31:32.887190       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:36:44.984293       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:38:35.345025       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:38:59.889811       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:45:48.986075       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:47:07.347719       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:48:01.892083       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:52:50.988789       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:54:33.893991       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:55:42.350540       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 15:57:53.990720       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:01:02.896320       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:02:45.352498       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:03:22.992534       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:09:25.354266       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:10:50.898184       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:12:18.994134       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:15:10.356116       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:18:55.900435       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:20:34.996190       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:21:53.358602       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:25:24.903159       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:27:08.361217       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:30:22.998966       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:30:49.905460       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:36:40.907886       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:37:03.363846       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:39:25.000753       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:45:03.366521       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:45:54.909899       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:46:02.003206       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:52:52.911824       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:53:13.368815       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 16:54:30.005752       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:00:31.007620       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:01:03.914429       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:02:10.371025       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:06:40.010059       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:07:26.916336       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:08:52.373063       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:13:28.012615       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:16:22.375600       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:16:49.918529       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:18:46.015193       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:22:58.920602       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:23:28.377770       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:27:35.018178       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:30:02.922752       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:30:17.380513       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:33:03.020415       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:37:40.383020       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:39:08.925358       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:41:10.023488       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:43:48.386117       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:46:44.928211       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:47:10.026058       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:49:36.388501       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:53:01.028723       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:53:09.930596       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 17:58:42.391286       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:01:19.031283       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:01:56.932112       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:04:43.394397       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:06:57.934114       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:07:43.033681       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:12:56.396390       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:13:46.936076       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:17:25.037103       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:21:08.938011       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:22:34.399097       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:26:09.040149       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:26:59.940799       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:32:09.401990       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:35:27.943246       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:35:58.042712       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:40:17.405597       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:42:22.945566       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:45:42.044739       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:46:14.408589       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:51:07.948188       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:53:52.411217       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:55:20.047566       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:56:30.950069       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 18:59:24.413513       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:02:09.050296       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:04:11.951723       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:06:34.416299       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:09:57.953576       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:10:25.053202       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:14:37.418555       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:18:05.055626       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:19:20.955959       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:23:26.420599       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:25:44.057626       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:26:39.958760       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:30:51.060168       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:31:01.423045       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:33:51.961489       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:37:01.062509       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:39:33.425186       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:40:26.963680       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:42:45.065229       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:48:40.427280       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:48:52.067479       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:50:10.965607       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:55:07.430226       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:56:40.967313       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 19:58:50.075527       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:00:11.432604       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:03:57.078605       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:05:13.969538       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:07:18.435002       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:10:36.081653       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:13:27.972176       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:15:40.437398       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:17:50.083285       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:20:35.974045       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:24:30.440009       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:27:07.086064       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:28:07.976711       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:29:51.445078       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:32:23.088090       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:36:50.447829       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:37:49.979466       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:38:52.090268       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:42:54.981933       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:43:14.449697       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:44:01.092758       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:49:04.094568       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:49:31.984255       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:49:36.452513       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:54:45.455163       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:56:38.096552       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 20:59:02.986412       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:00:53.457724       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:05:36.098844       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:06:06.460253       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:06:51.988753       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:12:55.100882       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:14:59.991617       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:15:26.462948       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:21:13.103155       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:22:50.994711       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:23:52.466031       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:28:40.996669       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:29:33.106367       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:32:50.468082       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:34:21.998948       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:38:15.108592       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:39:57.470364       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:43:21.001307       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:44:40.111222       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:49:15.472580       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:51:13.114132       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:51:34.003688       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:54:17.474798       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 21:58:47.006253       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:01:06.116702       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:02:51.476875       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:06:58.118879       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:08:07.008153       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:12:04.480155       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:13:04.121190       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:15:28.011064       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:19:07.123478       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:21:04.482101       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:21:33.012698       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:26:58.484958       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:27:26.014582       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:28:12.126373       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:33:39.017079       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:35:06.486961       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:37:24.129560       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:42:06.018656       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:42:21.489633       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:43:18.131852       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:47:38.021342       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:48:18.491864       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:50:01.134019       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:53:01.024261       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:55:16.135787       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:57:49.493858       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 22:58:28.026237       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:00:52.138241       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:06:42.496670       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:07:10.028109       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:10:38.140665       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:13:37.030490       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:14:34.499464       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:20:36.143004       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:21:09.033052       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:22:16.502504       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:25:58.144965       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:27:42.504601       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:29:46.035613       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:32:25.147274       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:34:46.037966       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:36:43.507240       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:40:41.150224       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:42:48.510377       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:43:05.040142       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:45:46.152695       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:48:04.512686       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:51:14.155151       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:51:33.042736       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:55:47.514754       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0506 23:58:14.157383       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:00:48.045273       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:03:23.160138       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:03:57.517598       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:07:42.048192       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:08:25.162191       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:12:18.520808       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:17:31.164046       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:17:39.050509       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:20:08.522704       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:22:41.166694       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:26:17.052552       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:28:19.525221       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:28:47.168835       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:32:12.054238       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:35:21.527545       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:37:13.171898       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:40:25.056572       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:40:57.530661       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:44:54.174682       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:48:08.532810       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:50:07.058680       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:53:46.177444       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:56:17.534975       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 00:59:31.061410       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:02:43.179956       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:04:33.537381       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:07:38.063957       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:08:52.183352       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:11:37.539714       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:14:35.066796       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:14:45.186287       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:17:54.542228       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:20:54.069452       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:23:02.188199       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:25:21.544350       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:27:25.072107       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:29:21.190797       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:32:33.546859       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:35:04.074735       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:37:40.193200       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:39:19.549168       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:43:48.077167       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:46:15.551892       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:46:44.195863       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:48:59.079133       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:52:32.198236       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:54:01.553965       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:58:13.081500       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 01:59:10.200586       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:03:47.556094       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:06:44.084464       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:08:09.203056       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:11:54.558702       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:13:43.087125       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:17:36.205126       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:18:50.561276       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:23:32.089604       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:23:51.563393       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:25:24.207863       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:28:49.092025       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:31:14.566578       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:31:41.209829       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:37:12.569031       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:38:15.094158       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:39:08.211877       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:43:42.571369       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:43:43.096326       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:47:44.214406       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:51:03.098988       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:51:04.574012       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:52:44.216819       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:57:40.575347       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 02:58:00.101897       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:00:19.219710       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:02:57.577741       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:04:40.104016       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:05:38.222255       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:11:10.580782       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:13:22.106891       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:15:19.225092       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:17:53.582530       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:21:06.227329       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:22:58.109946       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:24:27.584493       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:29:06.229780       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:32:34.586382       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:32:52.111883       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:34:54.232596       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:37:51.588618       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:42:47.115286       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:44:52.235175       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:46:27.590241       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:51:59.237354       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:52:36.118209       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 03:53:35.592172       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:00:44.121278       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:01:45.240436       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:03:20.594383       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:08:22.243136       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:09:01.124237       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:13:01.596758       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:14:24.245311       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:18:36.126814       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:21:02.598617       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:23:56.247348       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:26:02.600830       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:26:26.129000       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:32:30.249504       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:34:37.603844       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:36:25.130891       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:37:49.252188       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:40:44.605861       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:45:39.133642       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:47:07.254346       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:49:42.608739       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:54:33.135924       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:54:38.256774       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:56:53.611767       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 04:59:50.259067       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:01:58.614119       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:02:06.138556       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:09:15.262150       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:09:50.140506       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:11:17.616738       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:16:47.619306       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:17:32.264669       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:19:17.143732       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:22:41.621568       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:24:19.148674       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:24:48.267684       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:30:52.151129       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:31:27.623187       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:31:41.269878       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:35:52.152775       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:36:31.625863       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:41:02.272190       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:43:14.628346       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:44:33.158367       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:48:22.275009       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:49:23.630624       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:50:45.160549       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:55:24.633609       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:56:06.277773       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 05:59:44.163625       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:00:50.636248       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:02:46.280488       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:06:58.165733       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:10:07.638796       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:10:34.282624       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:12:41.167851       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:19:17.645318       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:19:37.284200       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:20:35.169592       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:25:04.648111       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:25:46.172285       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:28:40.286712       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:33:35.174824       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:33:46.650036       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:33:56.289488       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:40:42.177682       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:41:42.291830       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:43:07.652256       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:48:46.294433       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:48:57.655051       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:49:28.180037       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:56:38.297166       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:56:39.657587       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 06:58:54.182465       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:03:09.299283       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:03:43.658879       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:06:08.185216       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:12:07.302060       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:12:54.660851       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:13:21.187197       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:17:54.663629       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:20:43.303961       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:22:27.189224       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:27:08.666812       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:29:37.306616       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:30:53.191057       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:35:36.308714       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:36:26.669025       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:40:22.193319       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:44:35.672171       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:45:00.311715       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:47:52.196201       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:53:06.674212       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:54:05.314142       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 07:55:13.198964       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:00:25.677546       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:01:12.201934       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:01:53.316965       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:07:25.679878       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:10:51.204038       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:11:03.319617       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:14:11.681755       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:16:47.206520       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:18:55.322034       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:23:30.684191       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:25:08.208849       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:25:57.324052       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:29:07.686274       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:32:09.210972       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:35:31.326943       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:37:06.688856       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:40:11.213763       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:44:41.329044       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:45:28.691535       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:47:44.216448       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:51:55.331220       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:52:54.219295       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:53:11.694200       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 08:58:28.696122       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:01:52.333256       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:02:49.221854       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:05:51.698386       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:07:33.336160       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:12:16.700474       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:12:25.224192       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:14:28.338985       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:19:37.702842       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:20:24.226049       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:20:26.341158       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:28:26.228351       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:29:05.704852       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:29:32.343318       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:35:15.346201       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:36:17.707545       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:37:05.230325       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:43:46.348792       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:45:53.709934       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:46:27.232647       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:53:09.350990       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:53:26.712973       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:56:25.235088       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 09:58:32.353596       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:02:53.237218       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:02:53.715333       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:06:14.355860       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:08:04.717619       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:08:31.239209       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:15:11.241605       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:15:14.358138       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:18:01.721241       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:22:47.243684       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:23:55.359871       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:25:10.723623       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:29:51.246852       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:31:08.725829       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:31:59.362074       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:38:06.728185       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:39:26.364740       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:39:48.249484       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:45:06.367953       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:46:01.731233       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:49:07.251813       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:52:15.733940       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:53:59.370607       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:56:29.254284       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 10:57:19.736331       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:02:05.256947       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:02:13.372789       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:03:43.738322       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:09:48.374865       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:10:06.259414       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:10:26.741113       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:15:55.377437       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:17:12.261783       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:17:16.743861       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:23:41.746310       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:24:32.379571       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:24:55.264283       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:31:06.381862       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:31:29.748549       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:33:23.266881       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:36:07.384730       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:39:06.750859       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:39:34.269048       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:45:36.271225       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:45:42.386708       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:48:40.753292       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:52:24.273605       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:53:31.388630       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:55:17.755743       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 11:59:13.276199       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:01:08.390547       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:04:54.757953       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:07:54.277986       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:09:40.392964       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:12:36.759753       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:15:48.280171       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:18:40.762497       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:18:51.394982       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:25:20.765083       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:25:47.283779       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:26:03.397532       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:31:52.288166       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:32:36.767775       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:33:05.400063       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:37:15.290418       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:40:47.770396       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:42:01.401811       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:46:20.292692       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:48:29.772880       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:50:39.404253       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:51:57.294726       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:55:41.774850       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 12:56:14.406617       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:00:19.297421       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:02:56.777756       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:05:12.409165       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:07:35.299833       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:08:01.780179       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:10:51.411694       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:13:49.302041       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:15:53.782830       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:18:03.414581       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:20:24.304154       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:23:43.784674       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:25:27.416649       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:29:53.306819       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:31:28.787110       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:34:56.419894       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:37:11.789776       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:37:55.309469       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:40:19.422407       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:42:59.311756       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:44:30.792490       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:47:53.424359       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:50:16.794291       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:51:32.315984       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:54:12.426651       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:56:03.796871       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 13:59:44.318694       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:00:00.428587       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:01:44.799349       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:07:49.431368       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:08:47.320529       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:10:37.801068       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:14:17.434442       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:16:53.323295       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:18:54.803349       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:20:40.437072       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:24:05.806619       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:25:00.326510       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:26:24.439477       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:33:14.809507       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:34:34.443695       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:34:40.328738       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:38:53.811705       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:43:10.330587       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:43:47.445188       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:47:36.813845       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:51:31.333002       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:53:00.447809       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:56:23.816397       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 14:58:36.335850       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:02:06.449987       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:06:20.819023       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:07:51.338266       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:07:54.452696       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:14:28.821596       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:15:51.340640       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:17:10.454570       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:22:09.824455       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:22:51.343513       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:22:57.456390       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:28:04.458239       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:28:21.826418       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:28:39.345558       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:33:52.829133       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:34:46.348399       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:35:04.460689       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:42:50.351040       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:43:07.831411       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:44:43.462951       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:50:45.833789       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:51:03.464780       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:51:37.353735       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:57:50.356013       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 15:58:01.836401       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:00:47.467141       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:04:04.358700       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:07:59.838619       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:09:31.469497       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:11:05.360435       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:16:19.471851       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:17:08.840502       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:19:56.362486       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:22:46.474157       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:26:47.843343       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:28:14.475851       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:29:16.364498       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:36:27.846289       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:37:06.478354       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:38:09.366739       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:44:05.848461       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:45:06.480744       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:45:18.368963       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:49:18.851554       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:50:43.483979       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:54:25.371226       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:54:49.854068       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 16:56:43.486135       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:03:16.374208       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:03:28.856119       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:05:52.488122       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:09:41.858658       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:09:53.376809       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:10:52.490481       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:17:37.493078       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:17:58.861248       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:18:45.379404       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:25:08.381770       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:25:54.864003       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:26:06.494788       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:32:51.866848       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:34:24.384355       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:35:57.497573       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:39:35.869942       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:41:33.500253       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:42:56.387062       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:46:39.871886       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:48:49.389604       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:49:41.502410       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:53:07.874465       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:54:31.392021       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 17:56:45.504263       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:00:20.394321       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:02:36.877146       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:05:51.506771       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:08:18.397141       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:11:33.879518       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:12:57.508893       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:15:52.399879       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:18:18.881777       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:20:27.511039       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:22:08.402190       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:24:36.884104       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:26:15.513131       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:30:13.404766       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:31:56.515103       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:32:22.886238       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:36:21.406445       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:38:19.517329       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:40:51.888369       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:45:12.409033       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:47:07.519874       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:49:37.890527       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:51:15.411227       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:55:50.522764       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 18:58:23.893580       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:00:43.413547       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:02:13.524660       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:06:58.895980       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:08:01.526650       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:10:06.415604       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:15:02.528507       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:15:45.898636       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:19:38.417655       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:24:15.531225       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:25:21.901223       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:28:11.420320       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:32:02.533095       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:34:13.422254       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:34:51.903272       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:38:48.535100       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:39:28.424886       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:41:19.905458       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:46:50.427196       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:47:20.907401       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:48:04.537532       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:53:59.539486       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:54:28.429203       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:55:50.909649       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 19:59:50.431437       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:02:04.912666       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:02:48.541469       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:08:24.433959       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:10:08.543983       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:11:17.915431       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:14:57.436272       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:17:28.545698       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:20:05.918195       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:22:33.438410       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:24:18.547743       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:26:00.919940       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:29:38.550611       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:31:39.922037       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:31:46.442761       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:36:32.552705       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:40:19.444472       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:41:19.925114       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:46:20.554774       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:49:53.446359       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:50:57.927941       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:55:00.556862       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 20:57:31.448461       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:00:10.559508       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:00:41.929758       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:05:24.450965       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:07:44.931862       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:08:30.561629       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:13:17.453130       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:13:57.934613       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:14:52.563800       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:22:10.937201       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:22:26.454878       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:22:41.565948       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:30:01.457459       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:30:03.939979       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:31:06.568251       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:36:59.459644       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:37:07.941836       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:40:00.571037       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:44:01.943490       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:44:13.461988       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:46:40.573975       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:50:26.463757       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:53:20.945867       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:53:27.576158       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:58:35.466199       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:59:06.948115       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 21:59:39.578430       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:05:46.950646       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:08:10.468389       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:09:06.580080       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:12:07.953418       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:14:36.470212       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:18:58.956793       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:19:03.583309       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:23:05.472641       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:24:29.958985       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:24:44.585487       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:30:53.587809       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:32:00.474835       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:33:31.960809       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:37:19.477366       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:39:53.964036       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:40:06.590314       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:43:07.480155       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:45:36.966232       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:49:11.593154       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:51:22.968536       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:52:43.483119       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:55:08.595482       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 22:58:34.971117       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:00:06.484818       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:03:16.598286       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:04:51.973640       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:07:56.486823       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:11:41.975630       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:12:37.600965       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:16:14.489472       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:18:46.978164       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:19:15.603872       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:25:16.491792       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:25:35.606168       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:27:24.980428       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:32:02.494062       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:34:21.608077       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:35:59.982980       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:38:31.495746       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:42:15.609784       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:44:50.986078       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:45:34.498190       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:47:32.611986       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:52:00.501126       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:53:26.988969       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:56:59.615227       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0507 23:58:27.990589       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:01:40.503225       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:04:15.617738       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:04:44.993298       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:10:01.620107       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:11:06.505441       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:11:25.995703       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:17:23.507639       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:19:13.622595       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:20:49.998331       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:27:06.001128       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:27:12.510466       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:27:38.624206       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:33:08.512533       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:34:22.003445       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:36:15.626819       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:41:52.514866       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:43:58.006784       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:45:17.629344       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:50:33.517294       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:51:33.632135       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:51:41.009372       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 00:57:50.519457       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:00:00.011076       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:01:03.634555       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:04:35.523415       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:05:54.013968       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:08:06.636447       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:13:01.528104       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:14:38.016778       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:15:12.638536       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:20:24.530366       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:22:16.641223       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:22:17.019156       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:27:24.643042       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:27:50.532441       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:29:33.021353       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:32:52.534833       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:32:53.645040       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:38:30.023384       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:39:43.648245       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:42:49.537943       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:45:33.025092       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:47:45.650454       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:51:50.540606       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:55:28.027928       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:56:43.663890       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 01:58:51.542824       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:02:25.667097       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:04:00.030175       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:05:51.544707       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:09:18.033193       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:12:09.670043       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:13:50.546488       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:17:53.036074       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:20:48.673042       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:21:23.548200       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:27:10.675782       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:27:28.037882       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:28:23.550090       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:33:10.039859       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:33:35.678690       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:38:16.552671       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:41:25.680961       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:42:48.042290       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:46:21.555003       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:47:19.683472       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:52:45.045004       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:55:07.557481       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 02:56:24.686120       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:01:47.559738       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:01:53.047530       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:02:55.688800       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:08:15.049611       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:09:59.562750       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:11:14.690786       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:15:34.565005       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:17:37.052497       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:19:26.692899       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:24:32.695269       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:25:19.567275       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:26:11.055112       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:29:45.697979       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:34:30.569263       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:34:44.057173       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:38:10.700100       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:40:24.060094       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:41:21.572064       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:46:09.702870       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:47:00.062992       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:48:27.573718       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:52:34.705644       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:53:05.065586       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:55:19.575858       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:59:07.708093       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 03:59:33.067338       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:04:59.579595       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:06:51.709849       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:07:59.069256       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:11:54.582375       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:12:50.712614       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:14:05.071857       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:19:06.074721       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:21:50.584980       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:22:21.715204       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:27:28.077754       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:29:02.717434       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:31:36.587759       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:33:15.080355       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:35:21.719526       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:38:00.590230       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:42:29.082859       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:42:36.721491       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:45:56.592491       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:50:03.085257       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:51:08.723353       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:54:58.601540       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 04:57:48.087268       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:00:06.725442       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:02:00.604114       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:04:21.089259       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:09:02.728168       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:11:07.606244       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:12:56.092138       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:18:59.730685       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:20:00.609265       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:21:29.094556       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:26:43.734136       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:29:30.104136       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:29:46.611198       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:34:52.736864       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:37:42.106497       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:38:36.613484       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:40:18.739009       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:45:44.616529       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:46:17.108310       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:49:23.742242       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:52:16.618184       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:54:03.110506       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:55:45.744769       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 05:59:26.113174       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 06:01:06.621274       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 06:05:41.749378       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 06:05:52.114938       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 06:11:01.623184       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 06:15:15.117368       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 06:15:24.751741       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 06:20:04.627333       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 06:23:02.119282       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 06:23:56.753850       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 06:25:22.629234       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 06:28:57.760272       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 06:29:52.121882       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 06:34:11.638153       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 06:34:56.763094       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
E0508 06:35:16.124199       1 token_source.go:185] Unable to rotate token: failed to read token file "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory

Events in Namespace: istio-system

2m11s       Warning   FailedGetResourceMetric   horizontalpodautoscaler/cluster-local-gateway       failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
3m11s       Warning   FailedGetResourceMetric   horizontalpodautoscaler/istio-ingressgateway        failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
3m11s       Warning   FailedGetResourceMetric   horizontalpodautoscaler/istiod                      failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)

Events in Namespace: knative-serving

2m28s       Warning   FailedGetResourceMetric   horizontalpodautoscaler/activator   failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
2m28s       Warning   FailedGetResourceMetric   horizontalpodautoscaler/webhook     failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)

Events in Namespace: kubeflow-user-example-com

7m38s       Normal    Killing                 pod/pod-log-to-github-jt27k-system-dag-driver-490722581          Stopping container main
7m38s       Normal    Killing                 pod/pod-log-to-github-jt27k-system-dag-driver-490722581          Stopping container wait
21m         Warning   WorkflowFailed          workflow/pod-log-to-github-nm76f                                 
21m         Warning   WorkflowNodeFailed      workflow/pod-log-to-github-nm76f                                 Failed node pod-log-to-github-nm76f.root.upload-to-github
21m         Warning   WorkflowNodeFailed      workflow/pod-log-to-github-nm76f                                 Failed node pod-log-to-github-nm76f.root.upload-to-github.executor: Error (exit code 1)
21m         Warning   WorkflowNodeFailed      workflow/pod-log-to-github-nm76f                                 Failed node pod-log-to-github-nm76f
21m         Warning   WorkflowNodeFailed      workflow/pod-log-to-github-nm76f                                 Failed node pod-log-to-github-nm76f.root
39m         Warning   WorkflowFailed          workflow/pod-log-to-github-pm5rh                                 
39m         Warning   WorkflowNodeFailed      workflow/pod-log-to-github-pm5rh                                 Failed node pod-log-to-github-pm5rh.root.upload-to-github.executor: Error (exit code 1)
39m         Warning   WorkflowNodeFailed      workflow/pod-log-to-github-pm5rh                                 Failed node pod-log-to-github-pm5rh
39m         Warning   WorkflowNodeFailed      workflow/pod-log-to-github-pm5rh                                 Failed node pod-log-to-github-pm5rh.root.upload-to-github
39m         Warning   WorkflowNodeFailed      workflow/pod-log-to-github-pm5rh                                 Failed node pod-log-to-github-pm5rh.root
43m         Warning   WorkflowFailed          workflow/pod-log-to-github-scgtw                                 
43m         Warning   WorkflowNodeFailed      workflow/pod-log-to-github-scgtw                                 Failed node pod-log-to-github-scgtw.root
43m         Warning   WorkflowNodeFailed      workflow/pod-log-to-github-scgtw                                 Failed node pod-log-to-github-scgtw
43m         Warning   WorkflowNodeFailed      workflow/pod-log-to-github-scgtw                                 Failed node pod-log-to-github-scgtw.root.upload-to-github
43m         Warning   WorkflowNodeFailed      workflow/pod-log-to-github-scgtw                                 Failed node pod-log-to-github-scgtw.root.upload-to-github.executor: Error (exit code 1)
32m         Warning   WorkflowFailed          workflow/pod-log-to-github-tvwg6                                 
32m         Warning   WorkflowNodeFailed      workflow/pod-log-to-github-tvwg6                                 Failed node pod-log-to-github-tvwg6
36m         Warning   WorkflowNodeFailed      workflow/pod-log-to-github-tvwg6                                 Failed node pod-log-to-github-tvwg6.root.upload-to-github.executor: Error (exit code 1)
36m         Warning   WorkflowNodeFailed      workflow/pod-log-to-github-tvwg6                                 Failed node pod-log-to-github-tvwg6.root.upload-to-github
36m         Warning   WorkflowNodeFailed      workflow/pod-log-to-github-tvwg6                                 Failed node pod-log-to-github-tvwg6.root
32m         Warning   WorkflowNodeFailed      workflow/pod-log-to-github-tvwg6                                 Failed node pod-log-to-github-tvwg6.root-driver: Step exceeded its deadline
2m26s       Warning   WorkflowFailed          workflow/pod-log-to-github-z2xx6                                 
2m26s       Warning   WorkflowNodeFailed      workflow/pod-log-to-github-z2xx6                                 Failed node pod-log-to-github-z2xx6
2m26s       Warning   WorkflowNodeFailed      workflow/pod-log-to-github-z2xx6                                 Failed node pod-log-to-github-z2xx6.root.upload-to-github.executor: Error (exit code 1)
2m26s       Warning   WorkflowNodeFailed      workflow/pod-log-to-github-z2xx6                                 Failed node pod-log-to-github-z2xx6.root
2m26s       Warning   WorkflowNodeFailed      workflow/pod-log-to-github-z2xx6                                 Failed node pod-log-to-github-z2xx6.root.upload-to-github
3m47s       Normal    BackOff                 pod/pod-log-to-github-z9lb2-system-container-impl-2128329098     Back-off pulling image "python3.9"
53m         Warning   WorkflowFailed          workflow/pod-log-to-github-zj76w                                 
53m         Warning   WorkflowNodeFailed      workflow/pod-log-to-github-zj76w                                 Failed node pod-log-to-github-zj76w.root.upload-to-github.executor: Error (exit code 1)
53m         Warning   WorkflowNodeFailed      workflow/pod-log-to-github-zj76w                                 Failed node pod-log-to-github-zj76w
53m         Warning   WorkflowNodeFailed      workflow/pod-log-to-github-zj76w                                 Failed node pod-log-to-github-zj76w.root
53m         Warning   WorkflowNodeFailed      workflow/pod-log-to-github-zj76w                                 Failed node pod-log-to-github-zj76w.root.upload-to-github